acceptodds
Under review as a conference paper at ICLR 2027

Don’t Compress Away the Evidence: Secure Provenance Auditing for LLM KV Caches

Abstract

Efficient KV cache compression is essential for deploying long-context large language models, but it introduces an unexamined conflict with model ownership auditing. Existing fingerprinting methods embed ownership evidence through trigger patterns that are deliberately positioned away from routine task semantics—precisely the kind of patterns that cache compression discards as “unimportant". Therefore, we propose Fingerprint-Preserving Causal KV Cache Compression (FinCa), a framework that treats ownership evidence and task utility as joint constraints. Our key insight is to replace static trigger patterns with a low-rank perturbation in a KV projection that generates an input-conditioned structured signal in the runtime cache. Complementing this generation mechanism, we reserve fingerprint-bearing coordinates during compression and apply quantization-index modulation to protect them, ensuring that the ownership signal survives cache reconstruction. As a result, FinCa embeds a robust fingerprint signal from only 20 documents and achieves signal recovery in most evaluated settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Provenance, Ownership Auditing, Large Language Models, KV-Cache Compression, Causal Rate-Distortion Coding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QMRpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that trigger-based LLM fingerprints live in rare, semantically anomalous directions that KV-cache compression tends to discard, and proposes FinCa to make ownership evidence survive compression. FinCa (i) embeds a keyed rank-r perturbation into one Value projection so that ordinary inputs produce an input-conditioned signature in the cache, (ii) compresses the cache with a causal predictive codec that reserves the fingerprint subspace, allocates distortion via Fisher-weighted capped water-filling, and applies QIM on the reserved coordinates, and (iii) audits ownership by comparing reconstructed caches to a clean reference and running a Wald sequential test on calibrated H0/H1 score distributions. Experiments on ten open checkpoints report 0% BER and 100% exact recovery of a 2-symbol payload on 20 held-out WikiText-2 documents, with additional results under third-party cache policies, fine-tuning, and a comparison against seven prior fingerprinting methods.

Critical strengths

S1. A new and relevant angle. Treating inference-time KV-cache compression as a distortion channel for ownership evidence is, to my knowledge, not previously studied, and the idea of placing the fingerprint in the cache domain rather than in output behaviour is clean and well motivated.

S2. Careful audit design. The matched four-route protocol (C0/C1/F0/F1), the explicit calibration of QIM-induced bias under H0, and the pre-registered sequential test correctly separate weight-derived evidence from codec artifacts. This is more rigorous than most fingerprinting papers, which simply report trigger accuracy.

Critical weaknesses

W1. The threat model does not match the motivating threat, and this hollows out the main contribution. The introduction motivates the work with stolen weights redistributed by unauthorized parties, but the codec, authenticated cache traces, private decoder state, and QIM lattice are all executed by the deployer. A thief will not run the owner's fingerprint-preserving codec. The detection-constrained compression therefore only applies to a cooperative serving platform, where simpler attestation would suffice. Compounding this, the paper's own Appendix G shows the rank-2 perturbation already survives most third-party compression without the codec, so the problem the codec solves is largely absent for this signal.

W2. The headline comparison (Fig. 1, Table 5) is structurally unfair and partly undefined. FinCa keeps its carrier axes at 6 bits under its own codec while every baseline is pushed through KIVI-style INT2 plus eviction; the baselines are never given a protected subspace, so the table measures "protected vs. unprotected," not method vs. method. The adaptation of output-based methods (Trigger-set, UTF, EverTracer, DuFFin, LLMPrint) to "keys," "SNR," and "BER" in a KV channel is not explained; FinCa is listed at a 32x nominal factor though the main text reports approximately 3.2x; two FinCa rows (INT3/50% and INT2/50%) are numerically identical; and one entry reads Exact = 110.0%.

W3. The recovery results are close to vacuous because the payload is 2 bits, and the evaluation scale contradicts the long-context motivation. With r = 2, random guessing gives 25% exact recovery per document, so 0% BER across ten models says little about channel capacity; the ownership decision is actually carried by the continuous score and sequential test. The authors defer the payload-scaling study (where Phi conditioning degrades) to future work, but it is essential to the claim. All experiments use 96-token sequences and 20 audit documents, a regime in which no one compresses a KV cache, and the 20-document CIs (e.g., [83.2, 100]) are too wide to support "perfectly stable."

W4. Several reported numbers and protocol statements are internally inconsistent or implausible, which undermines trust in the empirical section. Clean WikiText-2 perplexities of ~1445 (Gemma 2 2B) and ~360 (Gemma 3 1B IT) are far above published values and suggest a tokenization/BOS bug, casting doubt on all Gemma delta-log-PPL claims. Min-SNR values of 1e5-1e7 arise from a near-zero variance denominator and are uninformative. Final-test Transfer ratios (3.06-3.82) exceed every search-trial value (<= 2.72) with no explanation. Section 4.2 says alpha, q, mu_max are fixed across models while Table 8 and Section 4.3 show them tuned per model. Table 6 promises FPR/TNR but reports only TPR/FNR. Robustness is also thin: fine-tuning erases the signal on Gemma 3 (10% TPR), and weight quantization, pruning, merging, and an adaptive low-rank-noise adversary are untested; no intrinsic baseline (e.g., REEF/CKA against the clean checkpoint) is provided under the same gray-box access.

Questions for authors
  1. In the stolen-weights scenario, who runs the FinCa codec? If no one, what role does Section 3.2 play in that scenario?

  2. How exactly are BER/SNR/Pass computed for the five LLM-fingerprinting baselines in Table 5, and how is FinCa run at 25% token retention given its codec has no eviction?

  3. Why do final-test Transfer ratios exceed all search-trial values?

  4. What are the clean, uncompressed perplexities of Gemma 2 2B and Gemma 3 1B IT under your pipeline, and how do they compare to published numbers?

  5. How do BER, Phi conditioning, and compression ratio change for r = 8/16/32 and for 2k-32k-token sequences?

Reviewer_fQs5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into an interesting conflict in LLM deployment: how inference-time KV-cache compression (quantization and token eviction) breaks existing model fingerprinting methods. The authors show that standard trigger-based or weight-watermarking signals often sit in low-energy directions or rely on rare token patterns that get wiped out when the KV cache is aggressively compressed. To address this, they propose FinCa, which embeds a keyed low-rank perturbation into a selected Value projection head and pairs it with a custom causal KV-cache codec. The codec uses an orthogonal transform, water-filling bit allocation, and Quantization-Index Modulation (QIM) to explicitly protect the fingerprint subspace during compression. Ownership is verified in a gray-box setting using least-squares recovery and a calibrated sequential probability ratio test (SPRT). Experiments across 10 open-weight models (0.6B to 9B) and several cache compression baselines (KIVI, H2O, KVQuant, etc.) show high recovery rates.

Critical strengths
  • Well-motivated problem: The observation that KV-cache compression acts as an unintentional signal-removal channel for model fingerprints is spot on. Table 5 and Appendix G clearly demonstrate that recent LLM fingerprinting methods (like UTF, EverTracer, and LLMPrint) degrade sharply once the cache goes down to INT4/INT2 with token eviction.

  • Solid mathematical formulation: The joint formulation of task distortion (weighted by Fisher sensitivity) and fingerprint SNR constraints via capped water-filling (Eq. 2 and Appendix C.3) is clean and well-thought-out. Using matched clean-compressed controls (C1C_1) to calibrate out the QIM-induced bias before running Wald's sequential test is also a very rigorous statistical design.

  • Broad empirical coverage: Evaluating across 10 different model architectures (including MLA-based DeepSeek-V2-Lite and GQA models) and re-implementing 7 recent fingerprinting baselines alongside 7 cache compression policies takes a lot of engineering effort. The layer-head propagation heatmaps (Fig. 4 and Appendix F) also give nice intuition on how Value perturbations travel downstream.

Critical weaknesses
  • Disconnect in the threat model (Who controls the codec?): This is my biggest conceptual struggle with the paper. FinCa requires a custom causal KV-cache codec with QIM and specific basis matrices to protect the fingerprint coordinates. If an adversary steals the model weights (like the LLaMA leak mentioned in the intro) and deploys them on their own infrastructure using standard vLLM, FlashAttention, or off-the-shelf KIVI/H2O, they have zero incentive to use the owner's custom FinCa codec. While Table 21 in Appendix G.2 shows that the weight perturbation alone survives H2O and KIVI reasonably well on some models, the core contribution in Section 3.2 (the causal codec + QIM) seems to assume a "trusted serving platform" where the auditor dictates the compression algorithm. The paper needs to be much more upfront in Section 2 about what real-world scenario allows gray-box access to internal projections/caches and forces the suspect deployment to run a custom codec.

  • Gray-box access requirements are very strong: In Section 3.3 and Appendix K, the auditor needs not only the reconstructed cache YY of the target head, but also the exact runtime hidden state inputs hth_t to the target projection layer to build the response matrix Φ\Phi, plus a clean reference model running on the exact same prompt. If an auditor already has layer-level activation/cache hooks into the suspect server, why wouldn't they just inspect the weights directly or run CKA on early-layer activations (like REEF) where KV-cache compression hasn't even happened yet? Tiny payload capacity (r=2r=2) and collision risk: All main experiments use a 2-bit payload (r=2r=2 physical symbols). With r=2r=2, there are only 4 possible sign vectors s∈{−1,+1}2s \in \{-1, +1\}^2 . While I understand the final ownership decision relies on the continuous score τ\tau via sequential testing rather than hard bit matching, calling a 2-bit signature a "fingerprint" is a bit of a stretch—it functions more like a zero-bit watermark (presence/absence detection). What happens to the conditioning of Φ\Phi and the PPL budget when rr scales to 32 or 64 bits to uniquely identify different downstream licensees?

  • Sequence length and serving overhead: The paper motivates KV-cache compression through long-context LLM serving, yet Appendix B notes that experiments are run on 96-token sequences from WikiText-2, running at roughly 3 to 20 tokens/sec on an A100 (Tables 9–18). Standard KV-cache compression barely kicks in at 96 tokens, and autoregressive scalar prediction (et=zt−μz^t−1e_t = z_t - \mu \hat{z}_{t-1}) across token steps introduces sequential dependencies that could bottleneck prefill parallelization and GPU kernel fusion.

  • Fragility under fine-tuning on certain models: In Table 6, while Llama-3.2 and Qwen3 hold up well under LoRA/SFT on Alpaca, Gemma-3 1B's TPR drops to 10% after 7 epochs on Chinese data. Why is the principal activation subspace VrV_r rotating or collapsing so much faster in Gemma-3 compared to Llama? A brief analysis of weight drift vs. carrier subspace alignment would strengthen Section 4.6.

Questions for authors

see weeknesses.

Reviewer_zsAMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Fingerprint-Preserving Causal KV Cache Compression (FinCa), a framework designed to preserve model fingerprints while compressing the KV cache. The key insight is to replace static trigger patterns with a low-rank perturbation in a KV projection, generating an input-conditioned, structured signal in the runtime cache. The reported results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper identifies a vulnerability associated with LLM KV-cache compression: compression can degrade fingerprint signals used to verify model provenance.
Critical weaknesses
  1. One of my main concerns is the practical relevance and scope of the problem. Do all fingerprinting methods embed their signals in locations that are unimportant to the model’s primary task? Which classes of fingerprinting methods are affected by KV-cache compression, and how significant is this problem in practice?

  2. In Table 5, FinCa achieves exactly the same results under INT3/50% and INT2/50%, including a median SNR of 1409 and a p10–p90 range of 770–2264. Could the authors clarify whether this is expected behavior or a reporting error?

  3. In Table 5, the exact recovery rate for Trigger-set under INT2/50% is reported as 110.0%, exceeding the maximum possible value of 100%. This appears to be a reporting error and requires correction.

  4. In Table 5, Trigger-set under INT4/full has a BER of 0.0% but an exact recovery rate of 78.25%. These values appear inconsistent if they are calculated over the same evaluation set. Could the authors explain this discrepancy?

  5. In line 427, the authors state that they report TPR and the corresponding FPR trajectories. However, Table 6 presents TPR and FNR, with no FPR results. Furthermore, the Chinese TPR for Gemma 3 1B IT after seven epochs of fine-tuning is only 10%, which appears inconsistent with a broad claim of robustness.

  6. Please improve the visual quality of Figure 2 by replacing the low-resolution images with high-quality vector graphics.

Given these reporting errors and unresolved inconsistencies, I have substantial concerns about the reliability of the reported results and recommend rejection in its current form.

Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.