acceptodds
Under review as a conference paper at ICLR 2027

CacheTok: One Binary State per Sampled Image Token, Read by Every Layer

Abstract

Block-autoregressive image generators such as VAR, Infinity, and PAR sample an image in a few blocks of tokens, yet every block attends to all earlier tokens, whose keys and values the transformer stores separately at each layer: Infinity-2B keeps 3.1 GiB of them for one image. Existing compressors for these models choose which tokens to keep, while later scales keep reading the oldest ones. We locate the redundancy along depth. Once a block is sampled, the per-layer keys and values of its tokens are views of one fixed content, and one linear subspace shared by all layers captures most of their energy. CacheTok writes each sampled token once, as sign bits per key–value head computed from its final hidden state and its token embedding, and every layer decodes its own keys and values from these bits through two linear readers; the losses of later blocks train the writer through a straight-through estimator. We prove that a shared state never needs more width than per-layer states and that exact training becomes sequential over blocks, a cost that grows with the number of blocks and stays small for next-scale generators with 10–13 blocks. On VAR-d16, PAR-XL4x, and Infinity-2B, CacheTok stores 512×, 1152×, and 1024× less visual history than matched full caches, changes FID by at most 0.03 and GenEval by −0.23 points, and doubles the batch capacity of Infinity-2B on one 80 GB GPU.

Primary area:
generative models
Keywords:
visual autoregressive models, binary history, attention cache, generative models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wtHMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CacheTok writes one binary state per sampled token and KV head from the final hidden state and sampled token embedding. Layer-specific linear readers reconstruct historical keys and values, while future-block prediction losses train the state through a straight-through estimator. Matched continuation experiments on VAR, PAR, and Infinity show small quality changes with 512–1152 times fewer persistent visual-history bytes. The paper analyzes the width advantage of shared states and the resulting sequential training dependency, and reports inference throughput and analytical batch capacity.

Critical strengths
  1. The matched Full-KV continuation baseline, multiple backbones, and repeated training seeds make the quality comparison considerably more informative than comparing only with untouched public checkpoints. FID changes of at most 0.03 in the primary study suggest that large reductions in stored history need not require deleting token positions.

  2. Removing either writer input, tying layer readers, or stopping state gradients degrades performance. Masking older positions also harms both Full-KV and CacheTok. Together, these controls support retaining sample-specific content and distinct layer readouts, giving the method a stronger basis than its compression ratio alone.

Critical weaknesses
  1. The paper omits VIAR: Visual Implicit Autoregressive Modeling, a relevant architectural alternative that replaces a deep explicit VAR stack with an implicit equilibrium layer and controls refinement across scales to reduce compute and KV usage.

  2. Appendix E labels the increase from 11 to 22 Infinity images as analytical capacity. Measured common-batch throughput improves by 1.46 times, but batch-one speed improves by only 1.05 times, and final-scale temporary tensors become the dominant memory term. Moreover, the method changes training from an L-stage to a KL-stage dependency. These distinctions matter when deciding whether the additional training and kernel complexity are justified by an experimentally demonstrated deployment advantage.

  3. Theorem 1 establishes an advantage for an unconstrained real-valued shared linear representation of a fixed cache matrix. It does not establish that a binary state produced by one writer shared across images has the same property, or that preserving cache energy preserves attention and generation quality after normalization and softmax. The empirical results support the implemented method, but the theory does not yet explain why the chosen bit width is sufficient. This distinction matters because the shared-subspace analysis is presented as a principal justification for the architecture.

  4. The controls do not fully isolate CacheTok's advantage over alternative compact learned histories. The paper discusses CLLA's shared latents with layer-specific projections, while Table 3 compares a same-width BF16 state whose payload is 16 times larger. Neither comparison answers whether a lower-dimensional continuous state or a 2-/4-bit shared state can achieve a better quality/latency trade-off at the same stored bytes. The external pruning configurations in Appendix C.3 also use different retention levels. A controlled comparison is needed to separate the benefits of depth sharing, sampled-token input, binarization, and the selected storage budget.

  5. Small differences in three-seed means do not establish quality equivalence. For example, the reported Infinity GenEval difference is -0.23 points, but its paired 95% interval is [-1.81, 1.35], which permits a substantially larger degradation than the headline mean. The larger-backbone extension adds useful evidence, but changes normalization and the continuation protocol. Thus, the evidence supports small observed average changes under the evaluated configurations more clearly than a tightly bounded quality-preservation claim for the default method.

Questions for authors

Is the 22-image Infinity batch executable on an 80GB A100? Please provide measured maximum batch, peak allocated memory, and throughput, together with the actual continuation-training cost relative to Full-KV.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wtHM (1/3): VIAR and the scope of Theorem 1

(1/3)

Thank you for a review that engages with the controls and the theory rather than the headline ratio. Your reading of the ablations, that keeping sample-specific content and distinct layer readouts gives the method "a stronger basis than its compression ratio alone", is the argument we wanted the paper to make, and your questions showed where it was still incomplete. Changes are blue in the revised PDF, and all numbers below refer to it.

W1. VIAR is now discussed (Sec. 2).

You are right that VIAR belongs in the paper. VIAR (Jiang et al., ICML 2026) replaces the deep middle stack of VAR with one implicit fixed-point layer and spends fewer iterations at large scales, which lowers parameters, compute, and memory. It targets the same depth factor of the cache as CacheTok, from the other side: VIAR redesigns and retrains the generator so that it has fewer effective layers, while CacheTok keeps a released generator with all its layers and changes only what is stored for each sampled token. The two compose: VIAR still keeps keys and values for every fixed-point iteration it runs, and one state per sampled token, read by every iteration through its own reader, is the same interface CacheTok gives to layers. The new sentence in Sec. 2 states the distinction.

W3. What Theorem 1 does and does not show.

We agree with the distinction, and the revision states it in a new "Scope of Theorem 1" paragraph (Sec. 3.2) and in the Limitations (Sec. 6). The theorem compares real-valued linear states fitted to one fixed history. It does not show that one binary writer shared across images reaches that optimum, and it bounds cache energy, not attention outputs after normalization and softmax (App. B.3 bounds a single attention operation). Its role is to motivate the structure, one state per position with one readout per layer; it does not fix the width b.

Two pieces of the gap are now covered.

  • Binary states. Prop. B.2 (App. B.2) gives binary states with fixed readers in which every layer's target has full rank b, so one b-bit code serves all L layers while lossless private codes need L times the bits. The advantage of one shared code over private codes therefore does not rely on real-valued states.
  • Learned states. App. G.5 (Table 21) measures what the trained writer actually stores. Linear readers fitted to the binary states recover 62.3% of the Full-KV history energy of Infinity-2B per history, against about 70% for the best real-valued rank-128 subspace, and 57.8% with readers shared across held-out histories. The writer, trained only on generation loss, extracts most of the shared subspace the theorem describes.

Energy is not the whole explanation, and the revision says so. A writer trained to reconstruct the Full-KV cache holds more energy (66.8%) yet generates 2.6 GenEval points worse (paired 95% CI [-3.75, -1.47]), because the cache never contains the token drawn at its own position, while the CacheTok state does.

On why b = d_h suffices: the theory does not decide it, and the text no longer suggests that it does. The evidence is empirical. The width sweep gives +4.20 GenEval from 64 to 128 bits and +0.26 [-0.53, 1.05] from 128 to 256 (Sec. 5.1), and the new equal-byte comparison (Table 18) shows that quality follows the number of stored coordinates, not the bits per coordinate.

(Continued in 2/3: deployment, training cost, and controlled comparisons.)

AuthorsOpenReview ↗

Response to Reviewer wtHM (2/3): measured capacity, training cost, equal-byte controls

(2/3)

W2 and your question. Does the 22-image batch run, and what does training cost?

It runs. App. E now reports a measured sweep on one A100 80GB SXM4 (Table 13), with the timing protocol of Table 12 and the largest batch that completes without running out of memory:

Infinity-2B, 1024²Full-KVCacheTok
Largest batch, analytic1122
Largest batch, measured1122
Peak allocated memory77.3 GiB75.8 GiB
Throughput at that batch0.257 img/s0.389 img/s

CacheTok fits 2.00× the images of Full-KV, and at each method's own largest batch it delivers 1.51× the throughput. We agree that batch-one speed (1.05×) is not a deployment argument; Sec. 5.2 and the Limitations now say that the benefit is memory first, and that once the history is small, the 2.7 GiB of final-scale temporaries bound capacity.

The training cost is measured in App. G.1 (Table 17; four A100s, BF16, one 10k-update continuation):

Span ratioTime per updateGPU-h per run (Full-KV → CacheTok)Peak memory
VAR-d16101.43×1.8 → 2.6+14%
PAR-XL4x14710.83×3.0 → 32.6+20%
Infinity-2B131.23×93.5 → 115.2+9%

On the next-scale models the dependent stages fall mostly in the small early blocks, while the late blocks that carry the work run at full width either way. PAR-XL4x is the expensive case, and App. G.6 shortens it: grouped producers or two- and three-pass training bring it to 2.15× to 3.84× Full-KV at FID 2.62 to 2.84 (exact: 2.65). The continuation is paid once per model, and the reader kernel is a FlashAttention-style tile loop with one extra small matrix product.

W4. A controlled comparison at the same stored bytes.

Thank you for spelling out the factors to separate; Sec. 5.5 and Table 5 are organized around them. Every row stores d_h bits per position and KV head and changes one factor (Infinity-2B GenEval; CacheTok 68.94):

  • Binarization vs lower-dimensional continuous or multi-bit states: 2-bit × d_h/2 65.62, 4-bit × d_h/4 62.58, BF16 × d_h/16 57.46. Quality follows the number of coordinates, so one bit per coordinate stores the most.
  • Sampled-token input and write position: writing from the block input before sampling, with CacheTok's bits, readers, and loss, changes GenEval by -3.61 [-4.82, -2.40].
  • Depth sharing with layer-specific projections (CLLA-style int4 latent): 65.81 at CacheTok's bytes, next to the pre-sampling row, so its gap comes from writing before sampling (Table 19).
  • Selected storage budget for pruning: Table 1 now lists each method's stored history (344 to 3240 MiB against 3.1). Continued with their compression active (Table 20), ScaleKV and FastVAR narrow but keep their gaps, and ScaleKV at a tenth of its budget, still 11× CacheTok's bytes, loses 4.3 points.

On latency at equal bytes: every row moves the same bytes per read, and lower-dimensional states only shrink the reader's multiply, which matters at batch one on VAR-d16 alone (0.238 vs 0.234 s for Full-KV). No row buys back the quality it loses.

(Continued in 3/3.)

AuthorsOpenReview ↗

Response to Reviewer wtHM (3/3): what three seeds support, and a summary

(3/3)

W5. Three-seed means do not establish equivalence.

You are right, and we have changed the claim rather than defend it. The abstract, Sec. 5.1, and the conclusion now speak of small average changes under the evaluated configurations, and Sec. 5.1 states that the three-seed GenEval interval [-1.81, 1.35] still admits a loss of up to 1.81 points.

To tighten the bound where you pointed, we continued Full-KV and CacheTok on Infinity-2B with two more seeds (71 and 101) under the same protocol (App. G.7, Table 23). The per-seed paired differences are -0.92, 0.33, -0.09, 0.16, and -0.44 GenEval points; with five seeds the mean difference is -0.19 and the 95% interval (four degrees of freedom) is [-0.81, 0.43]. The data now exclude an average loss larger than 0.81 points in this configuration. We present this as a bound on the average change, not as an equivalence test, because no margin was fixed in advance, and Table 1 keeps the three-seed means.

On the larger-backbone extension: we agree that its normalization and protocol differ, which is why App. F keeps it separate and labels the rows "CacheTok (ext.)". We cite it only as a second, independent setting with five seeds, where GenEval differs from Full-KV by -0.17 (Infinity-8B) and +0.09 (Lumina), not as evidence for the default configuration.

Summary of what changed in response to your review

  • VIAR is discussed in Sec. 2.
  • Capacity is measured, not only analytic: 11 vs 22 images on one A100, 1.51× throughput at those batches (Table 13).
  • Training cost is measured: 1.43×, 10.83×, 1.23× GPU time per update, with shorter schedules for PAR-XL4x (Tables 17 and 22).
  • Theorem 1 is scoped explicitly, and Table 21 measures how much of the shared subspace the learned bits hold.
  • Equal-byte controls separate depth sharing, the sampled-token input, binarization, and the budget (Tables 5, 18 to 20).
  • The quality claim is narrowed to small average changes, with a five-seed interval of [-0.81, 0.43].

Your review made the paper more careful in exactly the places where it had been too quick. If these changes resolve your concerns, we would be grateful if you would consider revisiting your score, and we are happy to answer any remaining question.

Reviewer_TTKCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the memory cost of historical KV (key–value) caches in block-autoregressive image generation, which grows with Transformer depth. Motivated by the observation that historical caches across layers share substantial low-rank structure, the authors propose CacheTok. It stores a shared binary state for each sampled token per KV head and reconstructs historical keys and values through layer-specific linear readers. Experiments on VAR (Visual Autoregressive Modeling), PAR (Parallelized Autoregressive Visual Generation), and Infinity report 512–1152× compression of the persistent visual-history cache while largely maintaining generation quality, together with improvements in inference throughput.

Critical strengths
  1. Meaningful problem and technical design The paper targets redundancy in historical caches across layers. By retaining all historical positions in a shared binary state, CacheTok substantially reduces cache storage while avoiding the direct loss of early visual information caused by token eviction. This is a meaningful design choice for visual generation, where earlier positions can remain relevant to later generation blocks.

  2. Comprehensive experimental validation The evaluation covers multiple image generation models and reports generation quality, cache size, inference latency, throughput, and ablations of key components. Overall, the experiments are reasonably comprehensive and support the conclusion that the method can achieve substantial compression of the visual-history cache.

Critical weaknesses
  1. The low-rank analysis does not substantiate the binary design. There is a lack of a direct link between the low-rank analysis and the actual binary cache construction. Figure 2 and Theorem 1 analyze the optimal real-valued representation for a fixed history, whereas the practical method employs a writer trained across histories that generates binary states from the final-layer hidden state and the sampled-token embedding. It is not guaranteed that the shared information identified in the former can be effectively extracted by the latter. While Figure 20 illustrates a correlation between captured cache energy and generation quality, it remains unclear whether the performance gains stem from extracting the shared structure of the original cache or from new history representations learned during joint fine-tuning. This limits the explanatory power of the theoretical analysis for the actual mechanism.
  2. The advance over prior work remains unclear. The technical contributions and their independent benefits relative to existing cross-layer sharing methods remain ambiguous. CLLA (Cross-Layer Latent Attention) [1] already uses shared latents, layer-specific projections, and quantization; similarly, xKV [2] has explored and leveraged the shared low-rank structure of KV (key-value) caches across layers. The primary distinctions of CacheTok lie in writing a binary state after sampling and training through later-block losses. Although Table 3 validates the role of these components within the current framework, the absence of comparisons against the most closely related methods under matched cache budgets and training conditions makes it difficult to quantify the incremental gains provided by these new designs over existing cross-layer sharing frameworks. [1] YANG Z, HAN J N, WU K, et al. Lossless KV Cache Compression to 2%[EB/OL]. arXiv:2410.15252, 2024. [2] CHANG C C, LIN W C, LIN C Y, et al. xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 12758–12778.
  3. The benefits and costs of training adaptation remain unclear. Full-KV receives the same continuation budget, but external compression plug-ins run on that backbone without compression-specific fine-tuning (Appendices C.3 and F.1). Comparisons with jointly fine-tuned CacheTok therefore do not isolate compression design from adaptation benefits. A baseline trained with compression under matched cache and training budgets would help separate these effects. Training time and peak training memory are also unreported despite the serial cross-block dependencies, while maximum batch capacity remains analytic. These gaps limit assessment of whether the cache savings justify training adaptation and additional reader computation.
Questions for authors
  1. Please provide evidence connecting the analyzed shared low-rank structure to the binary states learned from final-layer hidden states and sampled-token embeddings.
  2. Please compare with closely related cross-layer sharing methods under matched cache budgets and training conditions to isolate the gains from writing after sampling and training through later-block losses.
  3. Please include a baseline trained with compression under matched budgets to isolate adaptation gains, and report training costs and throughput at the maximum feasible batch size.
3 comments
AuthorsOpenReview ↗

Response to Reviewer TTKC (1/3): from the low-rank analysis to the learned binary states

(1/3)

Thank you for a precise review. You put your finger on the step the paper had left implicit, from the analysis of a fixed real-valued cache to a binary writer trained across histories, and you asked the right control questions about prior cross-layer work and adaptation. We also appreciate your reading of the design: keeping every historical position in a shared state, because early positions stay relevant to later blocks, is the point the masking study in Sec. 5.4 was built to show. Changes are blue in the revised PDF; numbers refer to it.

W1 / Q1. Does the trained writer extract the shared structure, or learn something new?

Both, and the revision now measures each part (App. G.5, Table 21; summarized in Sec. 3.2).

Setup. On 1065 DPG histories (one image per prompt, as in Figure 2) (Infinity-2B, 6425 positions before the final block), we teacher-force the seed-matched Full-KV model on each sample's tokens to get its cache C, and fit linear readers from the stored binary states Z to C at every layer. "Per history" fits readers to each history, so it measures how much of C lies in the column space of Z (upper bound: about 70%, the best rank-128 subspace in Figure 20a). "Shared" fits one set of readers on half the histories and tests on the other half, as CacheTok's own readers work.

Writer (b = 128)Energy, per historyEnergy, sharedGenEval
CacheTok62.3%57.8%68.94
No token input63.1%58.6%65.8
No hidden input38.4%33.9%64.3
Tied readers49.6%44.1%62.2
Stop state gradient45.3%40.2%61.2
Reconstruction-trained writer66.8%64.9%66.33

Three readings follow.

  1. The shared structure is extracted. With only the generation loss, the binary states hold 62.3% of the Full-KV energy, against about 70% for the best rank-128 subspace that any rank-128 real-valued state could hold. The structure found in Figure 2 is what the writer learns to store.
  2. Energy does not rank quality. The token-free writer holds slightly more energy and the reconstruction-trained writer the most, yet they generate 3.1 and 2.6 GenEval points worse. The hidden-free writer holds the least and still beats the tied-reader and stopped-gradient models.
  3. The extra content is the sampled token. A Full-KV key at position i is computed before the token at i is drawn (Sec. 4.1, Figure 4), so no amount of cache energy can contain it. The CacheTok state carries it, and the later-block losses decide how the b bits divide between the shared subspace and the token.

So the low-rank analysis explains why one state per position can serve all layers, and the token input together with the later-block losses explain why this state does better than a compressed copy of the old cache. The Figure 20(c) correlation you mention fits this picture: it compares widths of the same writer, where more coordinates help both terms at once. Sec. 3.2 now states the scope of Theorem 1 explicitly, including that it does not by itself justify the binary writer or its width.

(Continued in 2/3: CLLA, xKV, and the two new design choices at matched budgets.)

AuthorsOpenReview ↗

Response to Reviewer TTKC (2/3): CLLA, xKV, and the gains of each design choice

(2/3)

W2 / Q2. Comparison with CLLA and xKV under matched cache budgets and training.

You are right that Table 3 isolated the components only inside our own framework. The revision adds the comparison you asked for (Sec. 5.5, Table 5; App. G.3, Table 19). Every row in the upper block stores exactly CacheTok's bytes, d_h bits per position and KV head, and every learned row receives the same continuation (same data, 10k updates, three seeds) on the same backbones:

Infinity-2B, d_h bits/position/headVAR FIDGenEvalDPGSpan
CacheTok3.5868.9483.31KL
Pre-sampling write (CacheTok minus the late write)4.0665.3379.94L
Reconstruction-trained writer (CacheTok minus later-block loss)3.8766.3380.96KL
CLLA-style: shared int4 latent, per-layer projections3.9865.8180.27L
xKV-style, post-training cross-layer SVD15.8636.9258.40L
xKV-style + continuation5.3856.7370.38KL

Gain from writing after sampling. The pre-sampling row keeps CacheTok's bits, readers, and loss and only moves the write before sampling, which is where native caches and CLLA write. GenEval changes by -3.61 (paired 95% CI [-4.82, -2.40]) and VAR FID by +0.48.

Gain from training through later-block losses. Training the writer and readers to reproduce the Full-KV cache at every layer, the distillation alternative of Sec. 4.3, changes GenEval by -2.61 [-3.75, -1.47].

Both intervals exclude zero, so each of the two choices you identified as CacheTok's distinctions carries its own gain at this budget.

CLLA. At CacheTok's budget, the CLLA-style latent lands next to the pre-sampling row (65.81 vs 65.33), which places its gap on the write position rather than on int4 storage. We did not rerun CLLA at its own operating point; its authors report about 2% of the cache for language models, which would be 20× CacheTok's bytes here, and the final version will add that row. At equal bytes, its latent cannot contain the token about to be drawn.

xKV. xKV is a post-training method designed for up to 8× compression. At its design point (rank 4, 8×, 401.6 MiB) it stays within 0.7 GenEval points of Full-KV without training. At 1024× it collapses (36.92), and continuation recovers only 56.73, since eight BF16 coordinates per head are too few (compare the low-dimensional BF16 state in Table 18, 57.46).

The price. The pre-sampling and CLLA-style rows keep the native span L, while CacheTok pays KL. The measured cost of that is 1.43× to 10.83× GPU time per update (Table 17; more in 3/3). Sec. 2 now cites xKV and describes CLLA's shared latents with layer-specific projections as the closest prior design.

(Continued in 3/3: adaptation, training cost, and throughput at the maximum batch.)

AuthorsOpenReview ↗

Response to Reviewer TTKC (3/3): adaptation baseline, training cost, maximum-batch throughput

(3/3)

W3 / Q3. A baseline trained with compression, training costs, and throughput at the largest batch.

Compression-aware continuation of the plug-ins (App. G.4, Table 20). We repeated the continuation of the Full-KV backbone with the plug-in's compression active inside teacher forcing for the two plug-ins with the largest gaps in Table 1, ScaleKV and FastVAR, and for ScaleKV at one tenth of its budget, so the backbone adapts to the compression it meets at inference, with the same data, updates, and seeds as CacheTok:

Infinity-2B GenEval / VAR FIDas plug-incontinued with compressionstored history
ScaleKV (10%)67.7 / 3.7168.2 / 3.65344 MiB
FastVAR67.4 / 3.8367.9 / 3.72896 MiB
ScaleKV (1%)64.6 / 4.4134.9 MiB
CacheTok68.9 / 3.583.1 MiB

Adaptation explains part of the difference, as you expected: both gaps narrow. ScaleKV and FastVAR still trail CacheTok by 0.7 and 1.0 GenEval points, and the remainder follows how many positions each method removes (Sec. 5.4). These two are where adaptation has the most to recover; the final version extends the protocol to the other plug-ins. When token selection is pushed toward CacheTok's budget (ScaleKV at 1%, still 11× the bytes), it loses 4.3 points even after adaptation.

Training time and peak training memory (App. G.1, Table 17; four A100s, BF16). Time per update grows 1.43× on VAR-d16, 10.83× on PAR-XL4x, and 1.23× on Infinity-2B, against span ratios of 10, 147, and 13; peak memory per GPU grows by 14%, 20%, and 9%. One Infinity-2B run costs 115.2 GPU-hours against 93.5. For many-block generators, App. G.6 reduces PAR-XL4x to 2.15× to 3.84× Full-KV at FID 2.62 to 2.84 (exact: 2.65).

Maximum batch and throughput there (App. E, Table 13; one A100 80GB). The measured largest batch matches the analytic one on Infinity-2B: 11 images for Full-KV and 22 for CacheTok, at 0.257 and 0.389 images/s, 1.51×. On PAR-XL4x, CacheTok runs 235 images (analytic 237) at 6.207 images/s, against 146 at 5.040 for Full-KV.

Reader computation. Decoding costs b·d_h multiply-adds per historical key, and values are decoded once per query (Eq. 5). This is why VAR-d16 is 2% slower at batch one (0.238 vs 0.234 s) while every backbone is faster once memory limits the batch; the Limitations now say so.

Summary of the changes your review prompted: an energy analysis linking Theorem 1 to the learned bits (Table 21); matched-budget, matched-training comparisons with CLLA- and xKV-style histories that isolate the late write and the later-block loss (Tables 5 and 19); compression-aware continuation of the plug-ins (Table 20); measured training cost, peak memory, and maximum-batch throughput (Tables 13 and 17). If these close the gaps you identified, we would be grateful if you would reconsider your score, and we are glad to clarify anything further.

Reviewer_ZncTOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CacheTok, a key-value (KV) compression method for autoregressive image generation models. The authors observe inter-layer storage redundancy and propose storing only the KV pairs of attention heads, avoiding redundant load across layers. To further reduce the cache pressure caused by KV pairs, CacheTok proposes calculating a set of sign bits for each token position using a binary sequence and decoding the KV pairs for each layer using specific linear readers. The authors tested the method on various benchmark tasks, including class-to-image generation and text-to-image generation. Experimental results show that CacheTok can save significant cache space for multiple baseline models. For example, on an 80 GB A100 GPU, the paper estimates that the batch size for Infinity-2B can be increased from 11 images to 22 images. This paper presents novel ideas, well-developed experiments, and sufficient scalability and insights.

Critical strengths

Strengths

  • This paper employs binary states to store a shared subspace to reduce state storage and overhead. To this end, the authors further propose using the generation loss between each layer to constrain this binary space. This design not only reduces the loss caused by KV compression but also ensures the stability of each image part generation stage.

  • This paper points out that cross-layer KV storage in VAR models is redundant and does not need to be repeated for each layer. Furthermore, the authors propose and demonstrate the necessity of retaining the old token positions, supporting the rationale of this method.

  • CacheTok tests and verifies its effectiveness on multiple base models, including VAR-d16, PAR-XL4x, Inffnity-2B and even Lumina-mGPT-2.0 (Token-level AR model), indicating the scalability of the proposed method. In addition, the paper provides ample discussion and ablation experiments, thoroughly analyzing the impact of each component, and the experimental results are convincing.

Critical weaknesses

Weaknesses

  • Training scalability. Writing each state after the full Transformer pass introduces sequential dependencies across generation blocks. This results in a substantial training span for models with many blocks, such as PAR-XL4x (5,292 stages), potentially limiting scalability to other autoregressive generators.

  • Limited inference speedup. Despite substantial memory savings, inference acceleration is modest and depends on the batch size. For instance, CacheTok achieves only a 1.05× speedup over Full-KV for Infinity-2B at batch size one, suggesting that its primary benefit is memory efficiency rather than general inference acceleration.

  • Limited Quantitative Comparison of Compression Efficiency. CacheTok uses binary representations to compress KV pairs. Furthermore, CacheTok changes how KV pairs are stored across model layers, greatly reducing the memory usage burden. However, as shown in Table 1, CacheTok performs comparably to other KV selection methods, while the paper provides insufficient evaluation and comparison of storage efficiency to highlight the advantages of this approach.

Questions for authors
  1. Would the authors provide a more comprehensive comparison or statement of binary states with multi-bit quantization, such as 2-bit or 4-bit states, to better characterize the trade-off between memory efficiency and generation quality?
  2. Appendix section F.1 mentions that CacheTok is also applied to token-level AR models like Lumina-mGPT 2.0. Could the authors elaborate on how it's applied (since the paper seems to claim this method is specifically for VAR models)?
  3. Could the authors investigate potential strategies to reduce the training overhead of CacheTok, particularly for models with a large number of generation blocks or token-level autoregressive generators?
  4. I appreciate the observations and analysis of authors. Nevertheless, as mentioned in the weakness part, I believe the paper lacks further analysis and comparison of reducing memory usage capabilities for CacheTok. Therefore, could the author please provide more detailed comparative data with other KV compression methods? And i suggest revising the paper to highlight the efficiency advantages of the method.
  • I will definitely improve my score when my problems are solved.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZncT (1/3): training scalability and the token-level extension

(1/3)

Thank you for a warm and careful review. You describe the core of the paper exactly as we see it: the cross-layer history of a VAR model is redundant along depth, old token positions must still be kept, and one binary state written from the later-block losses can serve every layer. It means a great deal that you found the ideas novel and the experiments well developed, and that you checked the scalability claim across VAR-d16, PAR-XL4x, Infinity-2B, and the token-level Lumina-mGPT 2.0. Your questions asked for exactly the evidence the paper was missing; the answers below are all in the revised PDF (blue text, revised numbering).

W1 / Q3. Training scalability and how to reduce the overhead.

You are right that the late write serializes training over blocks. The revision now measures the cost instead of counting stages (App. G.1, Table 17; four A100s, BF16):

BlocksSpan ratioTime per update vs Full-KVPeak memory
VAR-d1610101.43×+14%
Infinity-2B13131.23×+9%
PAR-XL4x14714710.83×+20%

For next-scale generators the overhead is modest: the stages that wait for a previous write belong to the small early blocks, and the large late blocks, which carry most of the work, keep the GPUs busy either way. PAR-XL4x is the case you anticipated, so we tested the two strategies you asked about (App. G.6, Table 22):

PAR-XL4x scheduleStored factorSpanTime vs Full-KVFID
Exact (default)1152×529210.83×2.65
Four grouped producers, 64 bits each288×13503.84×2.62
Four groups, 16 bits each (packed)1152×13503.83×2.91
Three parallel passes1152×1083.18×2.73
Two parallel passes1152×722.15×2.84

Grouped producers (Prop. B.4) trade stored bits for span exactly and keep FID; few-pass training keeps the 1152× factor and loses a little FID, since it is exact only for the first T blocks. For generators with many blocks, these schedules bring training to a few times the cost of Full-KV.

Q2. How CacheTok is applied to Lumina-mGPT 2.0.

Thank you for asking; the original text was too brief, and App. F.1 now spells it out. Nothing in the interface is specific to next-scale prediction. On a token-level generator, each image token is its own block: after the token at position i is sampled, the writer maps the final hidden state at i and the embedding of the sampled token to a b-bit state per KV head, and every later token reads it through the per-layer readers inside the same softmax as its native entries. Text tokens keep their native keys and values. What changes with the generator is K, which for Lumina equals the number of image tokens, so exact training would be prohibitively serial. The Lumina continuation therefore uses the two-pass schedule above: the first pass teacher-forces all tokens in parallel with the native cache, the writer turns its final hidden states and the target tokens into states, and the second pass reads those states, with gradients through both passes. Sampling is unchanged. We present Lumina as a test of the interface together with this approximation, and the paper's training-cost claims stay with the next-scale models.

(Continued in 2/3: speed, storage efficiency, and multi-bit states.)

AuthorsOpenReview ↗

Response to Reviewer ZncT (2/3): inference speed, storage efficiency against other KV methods, multi-bit states

(2/3)

W2. Inference speedup is modest at batch one.

We agree, and the revision says so in Sec. 5.2 and in a new Limitations paragraph: the benefit is memory first, and at batch one CacheTok is 1.05× faster on Infinity-2B and slightly slower on VAR-d16. The speed gain appears once memory limits the batch. The new measured sweep (App. E, Table 13) shows this on real hardware: on one A100 80GB, Full-KV runs at most 11 Infinity-2B images and CacheTok 22 (peak allocated memory 77.3 and 75.8 GiB), and at these batches CacheTok delivers 0.389 against 0.257 images/s, 1.51×.

W3 / Q4. Storage efficiency against other KV compression methods.

Thank you for this suggestion; it changed how the paper presents its main comparison. You are right that Table 1 showed quality only, which made CacheTok look like one more method of similar quality. The revision adds the missing axis:

  • Table 1 now has a stored-history column for Infinity-2B. The methods within 0.3 GenEval points of CacheTok store 1790 to 3240 MiB per image (570× to 1030× more than CacheTok's 3.1 MiB), and ScaleKV, the most aggressive deletion, 344 MiB (110×) at a 1.2-point loss.
  • Table 13 measures what this buys on one GPU. The token-selection methods fit 10 (SparVAR) to 20 (ScaleKV) Infinity-2B images, against 22 for CacheTok. FastVAR fits 25, because it skips the two final scales whose temporaries bound CacheTok, at a cost of 1.5 GenEval points; the two reduce different terms of per-image memory and are complementary.
  • At comparable budgets, token selection breaks down. ScaleKV at 1% retention, still 11× CacheTok's bytes, loses 4.3 GenEval points even when the backbone is continued with that compression (Table 20).
  • Against cross-layer methods at CacheTok's bytes (Table 19), a CLLA-style int4 latent reaches 65.81 and an xKV-style factorization 56.73 GenEval, against 68.94 for CacheTok.

Sec. 5.5 opens with these equal-byte results, so the efficiency advantage is now stated in the main text.

Q1. Multi-bit states (2-bit, 4-bit).

Added in App. G.2 (Table 18) and Table 5. A q-bit state replaces the sign by a symmetric uniform quantizer rescaled to unit norm, trained with the same straight-through pass.

Bits × coordinatesBytes (Inf / VAR)VAR FIDInfinity GenEval
1 × d_h (default)16 / 83.5868.94
2 × d_h/216 / 83.8465.62
4 × d_h/416 / 84.3162.58
16 × d_h/16 (BF16)16 / 85.1757.46
16 × d_h (BF16)256 / 1283.5668.8

At a fixed budget, quality follows the number of coordinates, so one bit per coordinate is the most efficient choice. Extra precision at the default width changes nothing: the BF16 state with d_h coordinates matches the sign (Table 3).

(Continued in 3/3.)

AuthorsOpenReview ↗

Response to Reviewer ZncT (3/3): summary

(3/3)

Thank you again for engaging so constructively, and for saying you would raise your score once your questions were resolved. Here is where each one stands, in a form that may be convenient for the discussion with the AC.

  • Training scalability (W1, Q3). Measured cost: 1.43× and 1.23× GPU time per update on VAR-d16 and Infinity-2B, 10.83× on PAR-XL4x (Table 17). Grouped producers and few-pass training bring PAR-XL4x to 2.15× to 3.84× at FID 2.62 to 2.84 (Table 22).
  • Inference speed (W2). Memory first, stated in the Limitations; at the largest batch each method fits on one A100, CacheTok runs 22 vs 11 Infinity-2B images with 1.51× the throughput (Table 13).
  • Storage efficiency against other methods (W3, Q4). Table 1 now shows stored history: methods of similar quality keep 570× to 1030× more. At CacheTok's bytes, CLLA- and xKV-style histories lose 3.13 and 12.21 GenEval points (Tables 5 and 19).
  • Multi-bit states (Q1). At equal bytes, 2-bit and 4-bit states lose 3.32 and 6.36 GenEval points (Table 18).
  • Token-level models (Q2). The interface is generator-agnostic; App. F.1 now describes the Lumina writer, readers, and two-pass training.

Three points from the paper that the new results reinforce:

  1. Depth, not positions, is where the history is redundant. Masking old positions costs both Full-KV and CacheTok about 2.6 GenEval points (Table 4), while removing 99.8% or more of the stored bytes, and no position, costs 0.23.
  2. Each part of the write is needed. Writing before sampling costs -3.61 and training the writer to imitate the old cache costs -2.61 GenEval points at equal bytes, both with paired intervals that exclude zero (Table 5).
  3. The binary choice is the efficient one. At a fixed budget, coordinates matter and bits per coordinate do not (Table 18), which is why one sign per coordinate gives the 512× to 1152× reduction.

Your suggestion to put efficiency next to quality has made the main comparison of the paper much clearer. If the revision resolves your questions, we would be very grateful if you would consider raising your score, and we would be glad to answer anything else.

I sincerely appreciate the author's detailed response and the extensive supplementary experiments, which have substantially strengthened the reliability of this work. Nearly all of my concerns have been well addressed, and I have only one remaining suggestion: while CacheTok presents a novel and effective KV cache compression method, its description is not sufficiently clear. It is valuable to use clearer expressions and diagrams to illustrate the contribution of the method, especially in terms of efficiency.

I recommend accepting this paper and encouraging the author to carefully revise each section carefully.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): what changed in the revision

(1/2) General response to all reviewers and the AC

We thank the three reviewers for careful, generous reviews. Each of them pointed to the exact place where an argument was thinner than the text suggested, and the revision is better for it. New and revised passages in the PDF are in blue; table and section numbers below refer to the revised PDF. The new Table 5 shifts the submission's Table 5 and all later tables up by one.

What the reviewers found convincing. Reviewer wtHM notes that the matched Full-KV continuation, several backbones, and repeated training seeds make the quality comparison "considerably more informative than comparing only with untouched public checkpoints", and that the writer-input, tied-reader, and stop-gradient controls give the method "a stronger basis than its compression ratio alone". Reviewer TTKC calls keeping every historical position in a shared binary state "a meaningful design choice for visual generation", since early positions stay relevant to later blocks. Reviewer ZncT sees "novel ideas, well-developed experiments, and sufficient scalability and insights", and values the evidence that old token positions must be kept.

The shared questions. The reviews converge on four points: (a) whether CacheTok beats other compact histories (multi-bit, lower-dimensional, CLLA, xKV, adapted token selection) at the same stored bytes; (b) what the late write costs in training, in GPU time and memory rather than layer stages; (c) whether the batch doubling holds on real hardware; (d) how far Theorem 1 and the three-seed quality results reach. We ran the experiments each question needs, and where the evidence was narrower than our wording, we narrowed the wording.

Main changes.

  1. Equal stored bytes (Sec. 5.5, Table 5; App. G.2 to G.4, Tables 18 to 20): 2-, 4-, and 16-bit states, a pre-sampling write, a CLLA-style int4 latent, xKV-style factorization with and without continuation, a reconstruction-trained writer, and token-selection plug-ins continued with their compression active.
  2. Training cost (App. G.1, Table 17): measured time per update, GPU-hours, and peak training memory on all three backbones. App. G.6 (Table 22) adds grouped producers and two- or three-pass training for PAR-XL4x.
  3. Measured capacity (App. E, Table 13): largest batch, peak allocated memory, and throughput at that batch on one A100 80GB, and the stored history of every compared method; Table 1 gains a history column.
  4. Scope of Theorem 1 (Sec. 3.2) and what the learned bits hold (App. G.5, Table 21).
  5. Statistics (Sec. 5.1, App. G.7, Table 23): two more training seeds on Infinity-2B; the abstract now reports mean FID and GenEval changes, and Sec. 5.1 and Sec. 6 replace "matched quality" with "small average changes".
  6. Related work and scope (Sec. 2, App. F.1): VIAR and xKV are discussed; App. F.1 explains how CacheTok runs on the token-level Lumina-mGPT 2.0 and how its training handles the sequential dependency.
  7. Limitations (Sec. 6) and an updated AI-use statement.

The title is unchanged, no claim is extended, and every added experiment tests a claim the submission already made. The main text still ends on page 10. We also corrected transcription errors in a few table cells and the sentences that quote them. A summary of the new numbers follows in (2/2).

AuthorsOpenReview ↗

General response (2/2): new results at a glance

(2/2) New results at a glance

Infinity-2B GenEval unless stated; CacheTok is 68.94 and Full-KV 69.16 (three-seed means). Paired intervals are 95% t intervals over training seeds.

QuestionResultWhere
More bits per coordinate at the same bytes?2-bit × d_h/2: 65.62; 4-bit × d_h/4: 62.58; BF16 × d_h/16: 57.46. Quality follows the number of coordinates.Table 5, 18
Write before sampling, as native caches and CLLA do?-3.61 [-4.82, -2.40]Table 5, 19
Train the writer to reproduce Full-KV instead?-2.61 [-3.75, -1.47], although its states hold more Full-KV energy (66.8% vs 62.3%)Table 5, 21
CLLA / xKV at CacheTok's bytes65.81 / 56.73 (both continued); xKV at its own 8× design point stays within 0.7 of Full-KVTable 19
Plug-ins continued with compressionGaps narrow but stay open (ScaleKV 67.7 → 68.2, FastVAR 67.4 → 67.9); ScaleKV at 1% budget: 64.6Table 20
Training time per update vs Full-KV1.43× VAR-d16, 10.83× PAR-XL4x, 1.23× Infinity-2B (span ratios 10, 147, 13); at most 20% more peak memoryTable 17
Shorter schedules, PAR-XL4x2.15× to 3.84× Full-KV at FID 2.62 to 2.84 (exact: 2.65)Table 22
Does a 22-image batch run on one A100?Yes. Full-KV 11 images at 77.3 GiB, CacheTok 22 at 75.8 GiB; 1.51× throughput at these batchesTable 13
Learned bits vs Theorem 162.3% of Full-KV energy per history (best rank-128 subspace: about 70%)Table 21
Five training seedsPaired difference -0.19, interval [-0.81, 0.43] (three seeds: [-1.81, 1.35])Table 23

One result goes against an easy reading of the paper, and we report it as such: with the exact schedule, PAR-XL4x training costs 10.83× Full-KV per update. It is stated in Sec. 5.5 and App. G.1, and the Limitations paragraph collects it with the other costs.

We are grateful for the time the reviewers have given this paper and will answer any follow-up during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.