- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the reliability of cross-model KV cache reuse in multi-model LLM inference. The authors identify an important limitation of existing transferability metrics: reconstruction error alone does not reliably predict downstream performance, as the impact of cache errors depends on their alignment with the receiver's attention geometry. Based on a first-order analysis of attention perturbations, the paper introduces two complementary methods. Experiments show that CTI achieves a Spearman correlation of 0.91 with actual retention at substantially lower cost than fitting a mapper. CacheCert maintains empirical risk below its nominal target across the evaluated pairs while achieving meaningful prefill savings.
Critical strengths
-
The paper addresses an important challenge in multi-model inference. I particularly appreciate that the authors go beyond proposing another empirical metric and provide a principled explanation for why conventional reconstruction-based metrics can fail. The first-order attention analysis in Section 4.1 offers a clear conceptual insight: the downstream impact of cache errors depends not only on their magnitude but also on their alignment with the receiver's query-sensitive directions. The resulting sensitivity operator provides a meaningful foundation for the proposed transferability metric.
-
I find the overall framework elegant and well motivated. The paper connects its theoretical insights to two practical decisions: determining whether a model pair is suitable for transfer and deciding whether a particular transferred cache should be accepted or repaired. A particularly appealing aspect is the combination of an inexpensive, imperfect risk score with conformal risk control. The experiments are comprehensive, and the ablations generally support the proposed mechanisms. The improvements over reconstruction-based predictors and uncertainty-based runtime gates make the contributions convincing.
Critical weaknesses
-
Although the technical development is coherent, the presentation is relatively equation-heavy. In particular, Sections 4.1–4.3 introduce several mathematical quantities before providing an intuitive overview of how the complete framework operates. A high-level illustration explaining the relationship between attention sensitivity, offline CTI prediction, online self-consistency probes, and the fallback ladder would make the paper substantially easier to follow.
-
While CTI successfully identifies poorly transferable model pairs, the practical benefits of CacheCert become limited in these cases because a large fraction of requests must fall back to full prefill. For example, Gemma-12B → Qwen3-14B has a CTI of 0.13 and retention of 0.49, with only 8% of prefill work saved under CacheCert at the 10% risk target. However, a more detailed analysis of the underlying failure mechanisms would help clarify the practical scope of cross-model cache reuse.
Questions for authors
Q1. For poorly transferable pairs such as Gemma-12B → Qwen3-14B, could the authors provide a more detailed analysis of what drives the low CTI score? Relatedly, how are token positions aligned across model families with potentially different tokenizers?
Q2. The proposed sensitivity analysis relies on a first-order expansion of attention and subsequent approximations to the error operator. Could the authors further characterize the regime in which the approximation becomes unreliable?
Q3. Could the authors clarify whether the score quantiles used to construct the fallback ladder and the risk estimates used to select the conformal threshold are obtained from independent calibration subsets?
5 comments
Response to Reviewer LJbQ (Part 2)
(Following above Part 1)
- Placement, not magnitude. Its unweighted relative residual, 0.41, is barely above the reference pair's 0.31. Its exposure, the share of residual energy falling in the directions the receiver probes most relative to an isotropic residual, is 4.9 against 1.3. Substituting an isotropic channel metric raises the index from 0.13 to 0.47, so most of the penalty the index assigns is geometric.
- The key path carries the damage. Restricting the index to the value path leaves it at 0.74; restricting it to the key path leaves it at 0.09. This is the asymmetry Section 4.1 predicts, since the value path reaches the output without passing through the receiver's queries. We are careful to say it indicates where mapper capacity should go, not that a cheaper repair exists, since recomputing keys requires the forward pass that also produces values.
- A source-side architectural cause. Gemma-3's five-to-one interleaving of sliding-window and full-attention layers means most source rows summarize a bounded neighbourhood, which a globally attending receiver cannot use. Three things corroborate this: the two lowest indices in Table 8 both have Gemma as the source; the layer profile is flat because the deficit is spread over depth rather than localized, which is the mechanism Appendix C links to the collapse in work saved; and building source features only from Gemma's full-attention layers, a sixfold reduction in available source layers, raises the index from 0.13 to 0.29 and retention from 0.49 to 0.57. That partial recovery confirms the diagnosis while remaining far from a fix.
- What it is not. Head-geometry mismatch (256 on Gemma-3-12B against 128 on Qwen3-14B) only changes the width of and the conditioning of the ridge solve; sweeping the ridge parameter over two further orders of magnitude moves the index by less than 0.02. Cross-tokenizer pairing is ruled out by Table 5, where restricting to one-to-one blocks moves the index by at most 0.02. Neither accounts for a gap of this size.
Q2: The proposed sensitivity analysis relies on ...
A4: Yes. Corollary 1 already gave a checkable criterion in terms of the induced logit perturbation , but it is an upper bound and we only reported the median of per regime. We have now measured the quantity the bound bounds. For a sample of heads we compute the exact deviation by running the receiver on the perturbed cache and compare it against the first-order term, which gives the relative error head by head, together with the Spearman correlation between and the measured per-head damage. New Table 4:
| (nats) | share of head-query pairs | median relative error | Spearman of with damage |
|---|---|---|---|
| ≤ 0.05 | 0.34 | 0.031 | 0.96 |
| 0.05 to 0.15 | 0.41 | 0.072 | 0.93 |
| 0.15 to 0.30 | 0.16 | 0.148 | 0.88 |
| 0.30 to 0.60 | 0.07 | 0.287 | 0.79 |
| > 0.60 | 0.02 | 0.521 | 0.63 |
The regimes are therefore: below roughly 0.15 nats the expansion is accurate to within about a tenth and is close to a quantitative predictor; between 0.15 and 0.30 the magnitudes drift while the ordering survives; beyond 0.30 only the ordering is usable. The share of head-query mass above 0.30 nats is 0.03 on within-family pairs, 0.08 on cross-family pairs whose backbone holds up, and 0.34 on the three where it fails, so the regime in which the expansion is least reliable is the regime the index already flags as unsuitable.
We also document a second, independent way the criterion degrades, which the reviewer's phrasing about "subsequent approximations" prompted us to check: through the conditioning rather than through . Across receivers the median is 6.2 with a 95th percentile of 31, and the tail concentrates in the first two layers, where attention spreads nearly uniformly over a short prefix and loses rank. We exclude no heads but note that Corollary 1 is loose there.
Finally, on the approximations to the error operator specifically (A1 factorization, A2 dropped cross-term), the empirical validation in Appendix A.5 was already in the submission and is unchanged: per-head Pearson 0.981 against the exact Monte Carlo quantity, induced CTI rankings agreeing at Spearman 0.999, and accounting for a median 3.1% (within-family) and 7.8% (cross-family) of the total. We have kept Section 4.1's pointer to this short but made it explicit that the expansion is a local statement whose accuracy is governed by one measurable scalar. And, as before, none of this touches the online guarantee, which is derived in Appendix A.7 without any reference to the linearization.
Response to Reviewer LJbQ (Part 3)
(Following above Part 2)
Q3: Could the authors clarify whether the score quantiles used to construct the fallback ladder and the risk estimates used to select the conformal threshold are obtained from independent calibration subsets?
A5: Yes, they are independent, and Reviewer NJFU raised the same point, so we have treated it as a gap rather than a clarification. Concretely: two disjoint sets of 300 requests per workload, both disjoint from the test split, with supplying the score quantiles that define the rung boundaries and supplying the empirical risk in Eq. (3). This is now stated in Section 4.3, proved to suffice in new Proposition 7 (Appendix A.7, by conditioning on and applying the tower property), and specified with its cost in Appendix H, where the per-pair calibration figure rises from 9.2 to 10.5 GPU-minutes to reflect the second subset. We also report what reuse would have cost empirically (thresholds within one grid step, 0.006 of work saved, 0.003 of risk) while noting that only the split version is covered by the theory.
Response to Reviewer LJbQ (Part 1)
W1: Although the technical development is coherent, the presentation is relatively equation-heavy. ...
A1: Thank you, this is a good suggestion and we have adopted it. Section 4 now opens with Figure 1, which lays the four elements the reviewer names out left to right, and with a short roadmap paragraph that states the same thing in words before any new symbol is introduced.
The last of these is the connection the reviewer's question implies and the one the text alone made hard to see. The two lanes are not two methods but two readings of one quantity, and the dashed arrow says so visually: the same per-layer profile that screens a pair offline also fixes the order in which the ladder repairs layers online. The left-to-right arrangement was chosen so that the offline and online paths read as parallel tracks from a shared origin rather than as a hierarchy.
We also took the broader point about equation density into account while making the other revisions. Most of the detail added in response to the other reviews went into appendices rather than into Sections 4.1 to 4.3, and the copy-editing pass described in our reply to Reviewer 3g3P removed connective material from those sections, so the net change to the methodology section is small.
W2: While CTI successfully identifies poorly transferable model pairs ...
A2: We agree with both the observation and the request, and have added Appendix G on the three failed pairs, with a two-sentence pointer in Section 5.3. We also want to be straightforward about the framing: on these pairs CacheCert's contribution is to make the failure visible and priced rather than silent, and we do not present it as recovering useful savings there. The practical conclusion is that such pairs should not be deployed, which is what CTI is for.
The analysis itself is in A3 below. The part that bears most directly on "the practical scope of cross-model cache reuse" is the architectural finding: the failures are not generic cross-family effects but are driven by a specific property of the sender, namely that Gemma-3 interleaves sliding-window layers with full-attention layers in a five-to-one pattern, so most of the rows a Gemma sender offers summarize a bounded neighbourhood. This suggests the scope of cross-model reuse is better predicted by the sender's attention structure than by family labels, and it gives a concrete design rule: a globally attending sender can serve a sliding-window receiver more readily than the reverse.
Q1: For poorly transferable pairs such as Gemma-12B → Qwen3-14B ...
A3: On the second half of the question, token positions are never aligned: keys are mapped with rotary encoding stripped and re-rotated to the receiver's own positions at injection, so the correspondence is between content rows, derived from shared byte boundaries, and the two sequences may differ in length. The full protocol is A1/A4 to Reviewer NJFU and Appendix B.
On the first half, the new Appendix G decomposes the index. The headline is that the size of the residual is not what distinguishes these pairs, and a screening rule based on magnitude could not have separated them:
Four things follow for Gemma-12B → Qwen3-14B specifically.