- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TRACEKV is a test-time controller for reconstructing quantized KV-cache values while preserving the base quantizer's code alphabet, bit widths, group shapes, metadata count, and persistent storage layout. After prefill, it generates a short greedy BF16 continuation and replays it from an independently quantized cache to obtain value gradients of one-step and cumulative KL divergence. For each token, it chooses between independent quantization and quantization with residual feedback, using a signed gradient–error ledger and a gradient-based processing order. Completed candidate caches are evaluated and selected by replaying the same trace.
The paper evaluates the controller with affine quantization, KVarN, PatternKV, and OSCAR's rotation/clipping codec, and includes composition experiments with a port of SQuat's key quantizer. It reports reductions in teacher-forced cumulative distortion and improvements in free-running prefix retention on two frozen sources. Task accuracy improves on S1, but the improvement on S2 does not meet its prespecified accuracy gate. In the evaluated KVarN decode extension, TRACEKV reconstructs the prompt cache; newly completed 128-token decode blocks are quantized independently.
Critical strengths
-
A well-defined research question with a meaningful methodological contribution. The paper investigates whether KV-cache reconstruction can improve generation fidelity without changing the underlying quantization format, bit allocation, or retained token count. This isolates a meaningful degree of freedom complementary to existing quantization methods. The proposed signed error-feedback mechanism uses trajectory-derived gradients to guide reconstruction while preserving the base quantizer's persistent storage format and budget.
-
Careful mechanism validation and transparent experimental reporting. The signed-versus-magnitude ablations, fresh-source replication, and processing-order comparisons provide useful evidence for the proposed mechanism. Experiments across multiple quantization representations, together with evaluations beyond the selection horizon and SQuat composition results, further support its applicability. The paper also reports per-cell results, confidence intervals, and unsuccessful accuracy confirmations rather than presenting only favorable aggregate results.
Critical weaknesses
-
Unquantified construction overhead and unclear practical efficiency. TraceKV requires BF16 continuation generation, gradient computation, candidate construction, and repeated replay before committing the compressed cache. However, the paper does not report the corresponding construction latency or peak GPU memory. Preserving the persistent storage budget does not imply low construction overhead. Without measurements of time to first token, peak memory, and total generation latency, it is difficult to assess whether the fidelity improvements justify the additional computation compared with independent quantization.
-
Limited evidence for robust long-horizon and downstream task improvements. Although the paper evaluates beyond the selection horizon, the reported cumulative KL ratios do not isolate improvements in the subsequent tokens. For example, D128 includes the H64 selection window, so a lower cumulative distortion does not necessarily indicate continued improvement over tokens 65–128. Moreover, the free-running generation improvements remain modest in absolute terms, with mean R1024 values of 46.6 and 60.3 tokens, and the S2 task-accuracy improvement is not statistically significant. Most representation-level experiments also involve only a few prompt–model pairs. These results leave the robustness and practical significance of the trajectory-fidelity gains insufficiently established.
Questions for authors
Q1. Construction overhead and practical efficiency. Could the authors report the construction latency and peak GPU memory of BF16 trace generation, gradient computation, candidate construction, and replay selection? How do time to first token and total generation latency compare with independent quantization across different prompt and generation lengths? These measurements would help establish whether the fidelity improvements justify the additional computation.
Q2. Out-of-window effectiveness. Although the paper evaluates beyond the selection horizon, cumulative KL ratios do not isolate improvements on subsequent tokens. Could the authors report incremental KL over non-overlapping windows, such as D64 − D32 and D128 − D64? How does the benefit evolve during longer free-running generation, especially when newly generated KV blocks are quantized independently?
Q3. Robustness and downstream task performance. The S2 accuracy improvement is not statistically significant, and the results vary across models. Could the authors explain why improved BF16 prefix retention does not consistently translate into better task accuracy? Could they also evaluate more distinct prompts and report uncertainty estimates to establish the robustness of the observed gains?
Q4. Methodological choices and comparison with existing methods. The processing-order ablations do not consistently favor the default gradient-based path over temporal ordering. Could the authors clarify its advantages and provide sensitivity analyses for the feedback gain, block size, and selection threshold? Additionally, could they compare TraceKV with a suitable output-aware or sensitivity-aware method, such as OptR, at a comparable effective storage budget, with construction costs reported separately?
2 comments
Response to Reviewer jx5E (2/2): Q3–Q4
Q3. Robustness and downstream task performance
Prefix retention and answer correctness are related but not equivalent. Different wording or an alternative valid solution can break exact prefix agreement while preserving correctness. Conversely, closer agreement with an incorrect BF16 solution does not guarantee a correct answer.
To assess the accuracy gains more comprehensively, we expanded the evaluation from 100 questions to the complete MATH-500 dataset, preserving the original S2 experimental settings and scoring procedure.
| Model | BF16 | Independent KVarN | TraceKV | Improvement over independent |
|---|---|---|---|---|
| Qwen3-4B | 67.8% | 65.0% | 67.2% | +2.2 percentage points |
| DeepSeek-8B | 76.4% | 72.2% | 74.0% | +1.8 percentage points |
TraceKV improves observed accuracy in both models on this evaluation, producing 11 additional correct answers for Qwen and 9 for DeepSeek relative to independent quantization.
We also assessed fidelity across 56 distinct prompts covering mathematics, code, instruction following, and continuation:
| Model | Independent KVarN | TraceKV | Increase |
|---|---|---|---|
| Qwen3-4B | 15.96 | 26.82 | +10.86 |
| DeepSeek-8B | 15.25 | 24.88 | +9.63 |
Values are mean consecutive BF16-matching prediction positions in a diagnostic extending up to 256 steps, reported separately from the submitted R1024 metric.
These complementary evaluations show improved trajectory preservation across task types and higher observed answer accuracy on the complete mathematical benchmark.
Q4. Processing order, sensitivity, and comparison with OptR
Both gradient-based and temporal ordering are heuristics. Gradient ordering connects tokens with similar local sensitivities, while temporal neighbors can already have correlated sensitivities. Our method therefore does not require gradient ordering to be optimal on every input.
The core contribution is to exploit an additional degree of freedom: how cached values are reconstructed within a fixed quantization representation. Signed functional-error feedback accounts for cancellation and reinforcement across tokens, while trace-based selection chooses among completed candidates. Because the code alphabet, bit widths, and metadata layout remain unchanged, this approach is compatible with existing quantizers and complementary to their representation design.
We evaluated the main hyperparameters on 32 prompts per model:
| Parameter | Tested values | Main observation |
|---|---|---|
| Feedback gain | 0, 0.25, 0.5, 0.75, 1.0 | All tested nonzero gains improve construction-window KL; ratios are 0.568–0.672 for Qwen and 0.541–0.664 for DeepSeek. |
| Path block size | 16, 32, 64, 128 | Improvements persist across the tested sizes; ratios are 0.577–0.606 for Qwen and 0.583–0.621 for DeepSeek. |
| Selection tolerance | 1.00, 1.005, 1.01, 1.02, 1.05 | No selections change for DeepSeek, and at most one changes for Qwen. |
These results support stability across multiple parameter choices. Moderate feedback performs best in this sweep; larger gains can introduce stronger perturbations and worsen subsequent-window distortion.
Following your suggestion, we also compared against OptR using its public calibration optimizer. The comparison uses matched K2/V2 settings and the same effective KV storage budget, including retained full-precision tokens and metadata. On 32 evaluation prompts per model, adding TraceKV with its original gain of 0.5 gives:
| Model | TraceKV + OptR / OptR construction-window KL | Additional KL reduction | Construction time: OptR → TraceKV + OptR |
|---|---|---|---|
| Qwen3-4B | 0.867 | 13.3% | 0.098 → 7.958 s |
| DeepSeek-8B | 0.828 | 17.2% | 0.145 → 7.810 s |
Construction times include prefill. OptR calibration is reusable offline work; TraceKV construction is performed per input.
We then selected gain=0.25 using validation prompts and evaluated 32 additional prompt instances per model, retaining gain=0.5 as a matched control:
| Model | OptR | TraceKV + OptR, gain=0.5 | TraceKV + OptR, calibrated gain=0.25 | Calibrated improvement over OptR |
|---|---|---|---|---|
| Qwen3-4B | 40.72 | 50.16 | 50.47 | +9.75 |
| DeepSeek-8B | 54.19 | 62.00 | 70.13 | +15.94 |
Values are mean consecutive BF16-matching prediction positions, capped at 128. Gain is fixed before evaluation; there is no per-prompt gain search.
The additional gains are strongest for local KL and prefix retention, while later-window KL remains close to the OptR baseline. These results support complementarity: OptR improves the representation using calibration data, and TraceKV further improves reconstruction using the current input’s trajectory without increasing its persistent KV storage.
Response to Reviewer jx5E (1/2): Q1–Q2
Thank you for your careful review, because your four questions are clear, direct, and address the key issues in our work. We also appreciate how these questions have helped us think more deeply about our method and better understand its behavior.
Q1. Construction overhead and practical efficiency
We measured the complete construction cost, including trace generation, gradient computation, candidate reconstruction, and selection. The results for 1,024-token inputs are:
Measurements use an RTX 6000D, batch size 1, and BF16 eager QDQ. TTFT includes the entire controller, and peak memory includes model weights. The additional peak memory is relatively modest in this setting: 1.38 GiB for Qwen and 1.23 GiB for DeepSeek, while persistent KV storage remains unchanged.
The additional latency comes from computing input-specific sensitivities and evaluating candidate caches. This computation occurs only once per input. Subsequent decoding does not repeat tracing, backpropagation, or candidate selection; newly completed 128-token KV blocks are quantized independently.
Post-first-token decoding time therefore differs from independent quantization by approximately 0.2% or less. Across 1,024- and 2,048-token inputs, the total-time ratio decreases from 2.96–3.61× for 128 output tokens to 1.52–1.71× for 512 tokens. At 2,048 input tokens, TraceKV’s TTFT is 7.51 s for Qwen and 7.59 s for DeepSeek. These measurements identify a one-time startup cost whose relative contribution decreases with longer generation.
As a practical optimization, the 32 tokens generated for tracing can be streamed to the user as they are produced, reducing user-visible TTFT; overlapping trace generation with controller construction can also partially reduce the waiting time before token 33, although this remaining pause still requires further optimization.
Q2. Effectiveness beyond the selection window
To isolate the benefit beyond the construction window, we use only steps 1–32 for gradient computation, cache reconstruction, and candidate selection. We then separately evaluate incremental KL over steps 33–64 and 65–128, corresponding to D64−D32 and D128−D64.
Diagnostic setup: native chat templates, 24 mathematical prompts per model, and a shared greedy BF16 reference history. Values are geometric-mean ratios relative to independent KVarN; lower is better.
In this diagnostic, the benefit extends into the first subsequent window, reducing KL by 32.0% for Qwen and 14.2% for DeepSeek. Since this window excludes the construction steps, the improvement cannot be inherited from lower distortion within steps 1–32.
The benefit attenuates by steps 65–128. This is consistent with the finite-horizon design: the short trace captures local sensitivities, whose relevance can change as the generated context evolves. Nevertheless, the separate full MATH-500 evaluation below records accuracy gains of 2.2 and 1.8 percentage points under the original S2 settings, providing downstream benefits alongside the local fidelity improvements.