- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CacheTok writes one binary state per sampled token and KV head from the final hidden state and sampled token embedding. Layer-specific linear readers reconstruct historical keys and values, while future-block prediction losses train the state through a straight-through estimator. Matched continuation experiments on VAR, PAR, and Infinity show small quality changes with 512–1152 times fewer persistent visual-history bytes. The paper analyzes the width advantage of shared states and the resulting sequential training dependency, and reports inference throughput and analytical batch capacity.
Critical strengths
-
The matched Full-KV continuation baseline, multiple backbones, and repeated training seeds make the quality comparison considerably more informative than comparing only with untouched public checkpoints. FID changes of at most 0.03 in the primary study suggest that large reductions in stored history need not require deleting token positions.
-
Removing either writer input, tying layer readers, or stopping state gradients degrades performance. Masking older positions also harms both Full-KV and CacheTok. Together, these controls support retaining sample-specific content and distinct layer readouts, giving the method a stronger basis than its compression ratio alone.
Critical weaknesses
-
The paper omits VIAR: Visual Implicit Autoregressive Modeling, a relevant architectural alternative that replaces a deep explicit VAR stack with an implicit equilibrium layer and controls refinement across scales to reduce compute and KV usage.
-
Appendix E labels the increase from 11 to 22 Infinity images as analytical capacity. Measured common-batch throughput improves by 1.46 times, but batch-one speed improves by only 1.05 times, and final-scale temporary tensors become the dominant memory term. Moreover, the method changes training from an L-stage to a KL-stage dependency. These distinctions matter when deciding whether the additional training and kernel complexity are justified by an experimentally demonstrated deployment advantage.
-
Theorem 1 establishes an advantage for an unconstrained real-valued shared linear representation of a fixed cache matrix. It does not establish that a binary state produced by one writer shared across images has the same property, or that preserving cache energy preserves attention and generation quality after normalization and softmax. The empirical results support the implemented method, but the theory does not yet explain why the chosen bit width is sufficient. This distinction matters because the shared-subspace analysis is presented as a principal justification for the architecture.
-
The controls do not fully isolate CacheTok's advantage over alternative compact learned histories. The paper discusses CLLA's shared latents with layer-specific projections, while Table 3 compares a same-width BF16 state whose payload is 16 times larger. Neither comparison answers whether a lower-dimensional continuous state or a 2-/4-bit shared state can achieve a better quality/latency trade-off at the same stored bytes. The external pruning configurations in Appendix C.3 also use different retention levels. A controlled comparison is needed to separate the benefits of depth sharing, sampled-token input, binarization, and the selected storage budget.
-
Small differences in three-seed means do not establish quality equivalence. For example, the reported Infinity GenEval difference is -0.23 points, but its paired 95% interval is [-1.81, 1.35], which permits a substantially larger degradation than the headline mean. The larger-backbone extension adds useful evidence, but changes normalization and the continuation protocol. Thus, the evidence supports small observed average changes under the evaluated configurations more clearly than a tightly bounded quality-preservation claim for the default method.
Questions for authors
Is the 22-image Infinity batch executable on an 80GB A100? Please provide measured maximum batch, peak allocated memory, and throughput, together with the actual continuation-training cost relative to Full-KV.
3 comments
Response to Reviewer wtHM (2/3): measured capacity, training cost, equal-byte controls
(2/3)
W2 and your question. Does the 22-image batch run, and what does training cost?
It runs. App. E now reports a measured sweep on one A100 80GB SXM4 (Table 13), with the timing protocol of Table 12 and the largest batch that completes without running out of memory:
| Infinity-2B, 1024² | Full-KV | CacheTok |
|---|---|---|
| Largest batch, analytic | 11 | 22 |
| Largest batch, measured | 11 | 22 |
| Peak allocated memory | 77.3 GiB | 75.8 GiB |
| Throughput at that batch | 0.257 img/s | 0.389 img/s |
CacheTok fits 2.00× the images of Full-KV, and at each method's own largest batch it delivers 1.51× the throughput. We agree that batch-one speed (1.05×) is not a deployment argument; Sec. 5.2 and the Limitations now say that the benefit is memory first, and that once the history is small, the 2.7 GiB of final-scale temporaries bound capacity.
The training cost is measured in App. G.1 (Table 17; four A100s, BF16, one 10k-update continuation):
| Span ratio | Time per update | GPU-h per run (Full-KV → CacheTok) | Peak memory | |
|---|---|---|---|---|
| VAR-d16 | 10 | 1.43× | 1.8 → 2.6 | +14% |
| PAR-XL4x | 147 | 10.83× | 3.0 → 32.6 | +20% |
| Infinity-2B | 13 | 1.23× | 93.5 → 115.2 | +9% |
On the next-scale models the dependent stages fall mostly in the small early blocks, while the late blocks that carry the work run at full width either way. PAR-XL4x is the expensive case, and App. G.6 shortens it: grouped producers or two- and three-pass training bring it to 2.15× to 3.84× Full-KV at FID 2.62 to 2.84 (exact: 2.65). The continuation is paid once per model, and the reader kernel is a FlashAttention-style tile loop with one extra small matrix product.
W4. A controlled comparison at the same stored bytes.
Thank you for spelling out the factors to separate; Sec. 5.5 and Table 5 are organized around them. Every row stores d_h bits per position and KV head and changes one factor (Infinity-2B GenEval; CacheTok 68.94):
- Binarization vs lower-dimensional continuous or multi-bit states: 2-bit × d_h/2 65.62, 4-bit × d_h/4 62.58, BF16 × d_h/16 57.46. Quality follows the number of coordinates, so one bit per coordinate stores the most.
- Sampled-token input and write position: writing from the block input before sampling, with CacheTok's bits, readers, and loss, changes GenEval by -3.61 [-4.82, -2.40].
- Depth sharing with layer-specific projections (CLLA-style int4 latent): 65.81 at CacheTok's bytes, next to the pre-sampling row, so its gap comes from writing before sampling (Table 19).
- Selected storage budget for pruning: Table 1 now lists each method's stored history (344 to 3240 MiB against 3.1). Continued with their compression active (Table 20), ScaleKV and FastVAR narrow but keep their gaps, and ScaleKV at a tenth of its budget, still 11× CacheTok's bytes, loses 4.3 points.
On latency at equal bytes: every row moves the same bytes per read, and lower-dimensional states only shrink the reader's multiply, which matters at batch one on VAR-d16 alone (0.238 vs 0.234 s for Full-KV). No row buys back the quality it loses.
(Continued in 3/3.)
Response to Reviewer wtHM (3/3): what three seeds support, and a summary
(3/3)
W5. Three-seed means do not establish equivalence.
You are right, and we have changed the claim rather than defend it. The abstract, Sec. 5.1, and the conclusion now speak of small average changes under the evaluated configurations, and Sec. 5.1 states that the three-seed GenEval interval [-1.81, 1.35] still admits a loss of up to 1.81 points.
To tighten the bound where you pointed, we continued Full-KV and CacheTok on Infinity-2B with two more seeds (71 and 101) under the same protocol (App. G.7, Table 23). The per-seed paired differences are -0.92, 0.33, -0.09, 0.16, and -0.44 GenEval points; with five seeds the mean difference is -0.19 and the 95% interval (four degrees of freedom) is [-0.81, 0.43]. The data now exclude an average loss larger than 0.81 points in this configuration. We present this as a bound on the average change, not as an equivalence test, because no margin was fixed in advance, and Table 1 keeps the three-seed means.
On the larger-backbone extension: we agree that its normalization and protocol differ, which is why App. F keeps it separate and labels the rows "CacheTok (ext.)". We cite it only as a second, independent setting with five seeds, where GenEval differs from Full-KV by -0.17 (Infinity-8B) and +0.09 (Lumina), not as evidence for the default configuration.
Summary of what changed in response to your review
- VIAR is discussed in Sec. 2.
- Capacity is measured, not only analytic: 11 vs 22 images on one A100, 1.51× throughput at those batches (Table 13).
- Training cost is measured: 1.43×, 10.83×, 1.23× GPU time per update, with shorter schedules for PAR-XL4x (Tables 17 and 22).
- Theorem 1 is scoped explicitly, and Table 21 measures how much of the shared subspace the learned bits hold.
- Equal-byte controls separate depth sharing, the sampled-token input, binarization, and the budget (Tables 5, 18 to 20).
- The quality claim is narrowed to small average changes, with a five-seed interval of [-0.81, 0.43].
Your review made the paper more careful in exactly the places where it had been too quick. If these changes resolve your concerns, we would be grateful if you would consider revisiting your score, and we are happy to answer any remaining question.
Response to Reviewer wtHM (1/3): VIAR and the scope of Theorem 1
(1/3)
Thank you for a review that engages with the controls and the theory rather than the headline ratio. Your reading of the ablations, that keeping sample-specific content and distinct layer readouts gives the method "a stronger basis than its compression ratio alone", is the argument we wanted the paper to make, and your questions showed where it was still incomplete. Changes are blue in the revised PDF, and all numbers below refer to it.
W1. VIAR is now discussed (Sec. 2).
You are right that VIAR belongs in the paper. VIAR (Jiang et al., ICML 2026) replaces the deep middle stack of VAR with one implicit fixed-point layer and spends fewer iterations at large scales, which lowers parameters, compute, and memory. It targets the same depth factor of the cache as CacheTok, from the other side: VIAR redesigns and retrains the generator so that it has fewer effective layers, while CacheTok keeps a released generator with all its layers and changes only what is stored for each sampled token. The two compose: VIAR still keeps keys and values for every fixed-point iteration it runs, and one state per sampled token, read by every iteration through its own reader, is the same interface CacheTok gives to layers. The new sentence in Sec. 2 states the distinction.
W3. What Theorem 1 does and does not show.
We agree with the distinction, and the revision states it in a new "Scope of Theorem 1" paragraph (Sec. 3.2) and in the Limitations (Sec. 6). The theorem compares real-valued linear states fitted to one fixed history. It does not show that one binary writer shared across images reaches that optimum, and it bounds cache energy, not attention outputs after normalization and softmax (App. B.3 bounds a single attention operation). Its role is to motivate the structure, one state per position with one readout per layer; it does not fix the width b.
Two pieces of the gap are now covered.
Energy is not the whole explanation, and the revision says so. A writer trained to reconstruct the Full-KV cache holds more energy (66.8%) yet generates 2.6 GenEval points worse (paired 95% CI [-3.75, -1.47]), because the cache never contains the token drawn at its own position, while the CacheTok state does.
On why b = d_h suffices: the theory does not decide it, and the text no longer suggests that it does. The evidence is empirical. The width sweep gives +4.20 GenEval from 64 to 128 bits and +0.26 [-0.53, 1.05] from 128 to 256 (Sec. 5.1), and the new equal-byte comparison (Table 18) shows that quality follows the number of stored coordinates, not the bits per coordinate.
(Continued in 2/3: deployment, training cost, and controlled comparisons.)