acceptodds
Under review as a conference paper at ICLR 2027

Know Before You Transfer: Predicting and Certifying Cross-Model KV Cache Reuse

Abstract

Multi-model inference pipelines repeatedly process the same context when requests move between models. At each handoff, the receiving model typically prefills the shared context from scratch, even though the sender has already processed it. Existing methods reduce this cost by transferring the sender's key-value (KV) cache into the receiver's cache space through a learned or closed-form map. Their effectiveness, however, varies widely across model pairs, and failures are hard to anticipate or detect: a transferred cache may yield a fluent answer that disagrees with the output of a full prefill. Reliable deployment therefore needs inexpensive answers to two questions: which model pairs admit effective transfer, and how much risk a transferred cache poses for a given request. We address both through a first-order analysis of how cache perturbations propagate through attention. The analysis shows that downstream deviation depends on where the reconstruction error lies relative to the receiver's query geometry, a dependence that error magnitude alone does not capture. From it we derive the **Cache Transferability Index**, an offline diagnostic that predicts a pair's achievable retention from second-moment statistics alone, without fitting a mapper. We also develop **CacheCert**, an online procedure that turns the same geometry into a per-request risk score and applies conformal risk control to obtain a distribution-free bound on the rate of unsafe transfers under exchangeability. Across eighteen directed model pairs from three families and five long-context workloads, the index reaches a Spearman correlation of 0.91 with retention at roughly one fortieth the cost of fitting and evaluating a mapper. CacheCert keeps empirical risk at or below its nominal target on every pair. At a 10 percent risk budget, it saves 66 percent of prefill work on LongBench multi-hop QA and reduces silent failures from 14.7 percent to 4.1 percent on an agent-report track. Furthermore, both components are backbone-agnostic and can compose with existing transfer and repair methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache, LLM Inference Efficiency, Cross-Model Transfer, Conformal Risk Control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_LJbQOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the reliability of cross-model KV cache reuse in multi-model LLM inference. The authors identify an important limitation of existing transferability metrics: reconstruction error alone does not reliably predict downstream performance, as the impact of cache errors depends on their alignment with the receiver's attention geometry. Based on a first-order analysis of attention perturbations, the paper introduces two complementary methods. Experiments show that CTI achieves a Spearman correlation of 0.91 with actual retention at substantially lower cost than fitting a mapper. CacheCert maintains empirical risk below its nominal target across the evaluated pairs while achieving meaningful prefill savings.

Critical strengths
  1. The paper addresses an important challenge in multi-model inference. I particularly appreciate that the authors go beyond proposing another empirical metric and provide a principled explanation for why conventional reconstruction-based metrics can fail. The first-order attention analysis in Section 4.1 offers a clear conceptual insight: the downstream impact of cache errors depends not only on their magnitude but also on their alignment with the receiver's query-sensitive directions. The resulting sensitivity operator provides a meaningful foundation for the proposed transferability metric.

  2. I find the overall framework elegant and well motivated. The paper connects its theoretical insights to two practical decisions: determining whether a model pair is suitable for transfer and deciding whether a particular transferred cache should be accepted or repaired. A particularly appealing aspect is the combination of an inexpensive, imperfect risk score with conformal risk control. The experiments are comprehensive, and the ablations generally support the proposed mechanisms. The improvements over reconstruction-based predictors and uncertainty-based runtime gates make the contributions convincing.

Critical weaknesses
  1. Although the technical development is coherent, the presentation is relatively equation-heavy. In particular, Sections 4.1–4.3 introduce several mathematical quantities before providing an intuitive overview of how the complete framework operates. A high-level illustration explaining the relationship between attention sensitivity, offline CTI prediction, online self-consistency probes, and the fallback ladder would make the paper substantially easier to follow.

  2. While CTI successfully identifies poorly transferable model pairs, the practical benefits of CacheCert become limited in these cases because a large fraction of requests must fall back to full prefill. For example, Gemma-12B → Qwen3-14B has a CTI of 0.13 and retention of 0.49, with only 8% of prefill work saved under CacheCert at the 10% risk target. However, a more detailed analysis of the underlying failure mechanisms would help clarify the practical scope of cross-model cache reuse.

Questions for authors

Q1. For poorly transferable pairs such as Gemma-12B → Qwen3-14B, could the authors provide a more detailed analysis of what drives the low CTI score? Relatedly, how are token positions aligned across model families with potentially different tokenizers?

Q2. The proposed sensitivity analysis relies on a first-order expansion of attention and subsequent approximations to the error operator. Could the authors further characterize the regime in which the approximation becomes unreliable?

Q3. Could the authors clarify whether the score quantiles used to construct the fallback ladder and the risk estimates used to select the conformal threshold are obtained from independent calibration subsets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer LJbQ (Part 1)

W1: Although the technical development is coherent, the presentation is relatively equation-heavy. ...

A1: Thank you, this is a good suggestion and we have adopted it. Section 4 now opens with Figure 1, which lays the four elements the reviewer names out left to right, and with a short roadmap paragraph that states the same thing in words before any new symbol is introduced.

  • Left panel: the shared analysis. A single box, "First-order analysis of attention", holding the two objects everything else is built from, the per-head sensitivity operator and the per-layer damage profile, each with a small schematic inset beneath it.
  • Upper lane, reached by an arrow labelled "average over calibration corpus": offline, once per model pair. Cache Transferability Index, then "screen the model pairs", then a panel showing candidate pairs ranked and cut at a threshold.
  • Lower lane, reached by an arrow labelled "condition on live query": online, once per request. The per-request risk score, with its probe positions drawn as a strip of highlighted cache cells, then conformal risk control, with the calibrated cut shown on the score distribution, then the fallback ladder, four ascending rungs from "accept" to "dense prefill", each drawn with the number of layers it recomputes.
  • A dashed arrow leaves the damage-profile inset in the left panel, runs along the bottom of the figure and turns up into the ladder, labelled "same layer ordering".

The last of these is the connection the reviewer's question implies and the one the text alone made hard to see. The two lanes are not two methods but two readings of one quantity, and the dashed arrow says so visually: the same per-layer profile that screens a pair offline also fixes the order in which the ladder repairs layers online. The left-to-right arrangement was chosen so that the offline and online paths read as parallel tracks from a shared origin rather than as a hierarchy.

We also took the broader point about equation density into account while making the other revisions. Most of the detail added in response to the other reviews went into appendices rather than into Sections 4.1 to 4.3, and the copy-editing pass described in our reply to Reviewer 3g3P removed connective material from those sections, so the net change to the methodology section is small.

W2: While CTI successfully identifies poorly transferable model pairs ...

A2: We agree with both the observation and the request, and have added Appendix G on the three failed pairs, with a two-sentence pointer in Section 5.3. We also want to be straightforward about the framing: on these pairs CacheCert's contribution is to make the failure visible and priced rather than silent, and we do not present it as recovering useful savings there. The practical conclusion is that such pairs should not be deployed, which is what CTI is for.

The analysis itself is in A3 below. The part that bears most directly on "the practical scope of cross-model cache reuse" is the architectural finding: the failures are not generic cross-family effects but are driven by a specific property of the sender, namely that Gemma-3 interleaves sliding-window layers with full-attention layers in a five-to-one pattern, so most of the rows a Gemma sender offers summarize a bounded neighbourhood. This suggests the scope of cross-model reuse is better predicted by the sender's attention structure than by family labels, and it gives a concrete design rule: a globally attending sender can serve a sliding-window receiver more readily than the reverse.

Q1: For poorly transferable pairs such as Gemma-12B → Qwen3-14B ...

A3: On the second half of the question, token positions are never aligned: keys are mapped with rotary encoding stripped and re-rotated to the receiver's own positions at injection, so the correspondence is between content rows, derived from shared byte boundaries, and the two sequences may differ in length. The full protocol is A1/A4 to Reviewer NJFU and Appendix B.

On the first half, the new Appendix G decomposes the index. The headline is that the size of the residual is not what distinguishes these pairs, and a screening rule based on magnitude could not have separated them:

PairCTIvalue path onlykey path onlyisotropic G\mathbf{G}uniform wwrel. residualexposuretop-4 share
Qwen3 14B → 32B (reference)0.880.930.860.840.860.311.30.38
Qwen3 8B → Gemma 12B †0.440.780.380.620.520.393.40.14
Gemma 12B → Qwen3 14B †0.130.740.090.470.240.414.90.13
Gemma 4B → Llama 8B †0.160.710.110.490.260.444.60.12

Four things follow for Gemma-12B → Qwen3-14B specifically.

AuthorsOpenReview ↗

Response to Reviewer LJbQ (Part 2)

(Following above Part 1)

  1. Placement, not magnitude. Its unweighted relative residual, 0.41, is barely above the reference pair's 0.31. Its exposure, the share of residual energy falling in the directions the receiver probes most relative to an isotropic residual, is 4.9 against 1.3. Substituting an isotropic channel metric raises the index from 0.13 to 0.47, so most of the penalty the index assigns is geometric.
  2. The key path carries the damage. Restricting the index to the value path leaves it at 0.74; restricting it to the key path leaves it at 0.09. This is the asymmetry Section 4.1 predicts, since the value path reaches the output without passing through the receiver's queries. We are careful to say it indicates where mapper capacity should go, not that a cheaper repair exists, since recomputing keys requires the forward pass that also produces values.
  3. A source-side architectural cause. Gemma-3's five-to-one interleaving of sliding-window and full-attention layers means most source rows summarize a bounded neighbourhood, which a globally attending receiver cannot use. Three things corroborate this: the two lowest indices in Table 8 both have Gemma as the source; the layer profile is flat because the deficit is spread over depth rather than localized, which is the mechanism Appendix C links to the collapse in work saved; and building source features only from Gemma's full-attention layers, a sixfold reduction in available source layers, raises the index from 0.13 to 0.29 and retention from 0.49 to 0.57. That partial recovery confirms the diagnosis while remaining far from a fix.
  4. What it is not. Head-geometry mismatch (256 on Gemma-3-12B against 128 on Qwen3-14B) only changes the width of x\mathbf{x} and the conditioning of the ridge solve; sweeping the ridge parameter over two further orders of magnitude moves the index by less than 0.02. Cross-tokenizer pairing is ruled out by Table 5, where restricting to one-to-one blocks moves the index by at most 0.02. Neither accounts for a gap of this size.

Q2: The proposed sensitivity analysis relies on ...

A4: Yes. Corollary 1 already gave a checkable criterion in terms of the induced logit perturbation ∥Δs(q)∥2\lVert\Delta\mathbf{s}(\mathbf{q})\rVert_{2}, but it is an upper bound and we only reported the median of δ\delta per regime. We have now measured the quantity the bound bounds. For a sample of heads we compute the exact deviation Δo(q)\Delta\mathbf{o}(\mathbf{q}) by running the receiver on the perturbed cache and compare it against the first-order term, which gives the relative error head by head, together with the Spearman correlation between ρ\rho and the measured per-head damage. New Table 4:

∥Δs(q)∥2\lVert\Delta\mathbf{s}(\mathbf{q})\rVert_{2} (nats)share of head-query pairsmedian relative errorSpearman of ρ\rho with damage
≤ 0.050.340.0310.96
0.05 to 0.150.410.0720.93
0.15 to 0.300.160.1480.88
0.30 to 0.600.070.2870.79
> 0.600.020.5210.63

The regimes are therefore: below roughly 0.15 nats the expansion is accurate to within about a tenth and ρ\rho is close to a quantitative predictor; between 0.15 and 0.30 the magnitudes drift while the ordering survives; beyond 0.30 only the ordering is usable. The share of head-query mass above 0.30 nats is 0.03 on within-family pairs, 0.08 on cross-family pairs whose backbone holds up, and 0.34 on the three where it fails, so the regime in which the expansion is least reliable is the regime the index already flags as unsuitable.

We also document a second, independent way the criterion degrades, which the reviewer's phrasing about "subsequent approximations" prompted us to check: through the conditioning ς(q)\varsigma(\mathbf{q}) rather than through δ\delta. Across receivers the median ς\varsigma is 6.2 with a 95th percentile of 31, and the tail concentrates in the first two layers, where attention spreads nearly uniformly over a short prefix and V⊤Ψ(q)\mathbf{V}^{\top}\boldsymbol{\Psi}(\mathbf{q}) loses rank. We exclude no heads but note that Corollary 1 is loose there.

Finally, on the approximations to the error operator specifically (A1 factorization, A2 dropped cross-term), the empirical validation in Appendix A.5 was already in the submission and is unchanged: per-head Pearson 0.981 against the exact Monte Carlo quantity, induced CTI rankings agreeing at Spearman 0.999, and ∣X∣\lvert\mathcal{X}\rvert accounting for a median 3.1% (within-family) and 7.8% (cross-family) of the total. We have kept Section 4.1's pointer to this short but made it explicit that the expansion is a local statement whose accuracy is governed by one measurable scalar. And, as before, none of this touches the online guarantee, which is derived in Appendix A.7 without any reference to the linearization.

AuthorsOpenReview ↗

Response to Reviewer LJbQ (Part 3)

(Following above Part 2)

Q3: Could the authors clarify whether the score quantiles used to construct the fallback ladder and the risk estimates used to select the conformal threshold are obtained from independent calibration subsets?

A5: Yes, they are independent, and Reviewer NJFU raised the same point, so we have treated it as a gap rather than a clarification. Concretely: two disjoint sets of 300 requests per workload, both disjoint from the test split, with Dq\mathcal{D}_{\mathrm{q}} supplying the score quantiles that define the rung boundaries and Dcal\mathcal{D}_{\mathrm{cal}} supplying the empirical risk in Eq. (3). This is now stated in Section 4.3, proved to suffice in new Proposition 7 (Appendix A.7, by conditioning on Dq\mathcal{D}_{\mathrm{q}} and applying the tower property), and specified with its cost in Appendix H, where the per-pair calibration figure rises from 9.2 to 10.5 GPU-minutes to reflect the second subset. We also report what reuse would have cost empirically (thresholds within one grid step, 0.006 of work saved, 0.003 of risk) while noting that only the split version is covered by the theory.

Reviewer_NJFUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies two issues in cross model KV cache reuse: how to predict the portability of model pairs before fitting the mapper, and how to control the cache migration risk of a single request during reasoning. Through the first order perturbation analysis of attention, the author constructs the cache portability index CTI, which considers the query geometry of the receiving model, and proposes CacheCert. Combining a small amount of location detection, hierarchical recalculation and conformal risk control, the author constrains the divergence rate of the output relatively complete pre filling results under the exchangeability assumption. The prediction ability and computational savings are verified on 18 directed model pairs and five types of workloads.

Critical strengths
  1. The paper focuses on the problem that the effect of cross model cache migration is difficult to predict and the failure is difficult to detect, and provides offline filtering and online control mechanisms. Compared with only improving the average migration quality, this design more directly responds to the deployment decision requirements in multi model reasoning.

  2. Attention disturbance analysis reveals the effect of query direction, attention distribution and inter layer propagation on error, and designs CTI and request level scores accordingly. The ablation results show that the removal of sensitivity weighting or query conditionality will significantly reduce the prediction ability or calculation income, providing support for the core design.

  3. The experiment covers multiple model families, migration backbones, and long context tasks, and reports on failed model pairs, workload migration, and non monotonic repair. The author also makes it clear that the consistency of the output for full pre filling is guaranteed, rather than the factual correctness of the answer.

Critical weaknesses
  1. Methods Use paired cache rows to estimate covariance and compare migration caches at corresponding locations, but the segmentation results of different model families usually cannot be directly mapped line by line. The paper lists word segmentation mismatch as future work, but reports six groups of cross family experiments, which need to explain the actual alignment, shared context representation and location coding processing methods.
  2. The appendix uses the empirical quantile of the calibration score to construct the threshold, and then uses the calibration loss to select the control parameters. If the same batch of samples are used in two steps, the loss function family itself also depends on the calibration data, and the standard conformal risk control conclusion cannot be directly confirmed to be applicable only by the loss monotonicity. The author needs to explain whether to use independent data partition, or provide proof to override the data dependency structure.
  3. The 18 model pairs share multiple source models and receiving models, and the overall high correlation may partly come from family differences, rather than the stable prediction ability of the new model combination. Cross family correlation is only based on six model pairs. It is suggested to add reserved models or reserved family tests, and report intra group correlations and confidence intervals grouped by models to enhance the evidence of CTI for screening unknown models.
Questions for authors
  1. How do six cross family experiments deal with the differences in token boundaries and sequence lengths generated by different word breakers? How are cache rows, probe locations, and paired covariance aligned?

  2. Are independent samples used to construct empirical quantile thresholds and perform conformal risk calibration? If the same batch of samples is reused, which theoretical result guarantees that the data dependence loss family still satisfies equation (12)?

  3. After reserving a certain source model, receiving model or the whole model family, what is the prediction relevance of CTI? How much of the overall correlation improvement comes from the performance difference inside and outside the family?

  4. After the source model attention statistics, online detection and repair are all included in the cost, what are the end-to-end first token delay benefits under different context lengths?

7 comments
AuthorsOpenReview ↗

Response to Reviewer NJFU (Part 1)

W1: Methods Use paired cache rows to estimate covariance and compare migration caches at corresponding locations, but the segmentation results of different model families usually cannot be directly mapped line by line ...

A1: Thanks for your criticism: we ran six cross-family experiments while listing tokenizer mismatch as future work, without stating what we actually did in between. We did have a concrete protocol; we failed to describe it, and the future-work item was scoped too broadly. Both are now fixed, with a short paragraph in Section 3 and the full specification in the new Appendix B.

The pairing we use. All tokenizers involved segment a context string into contiguous byte spans, whether by byte-level merges (Qwen3, Llama-3) or by a unigram model with byte fallback (Gemma-3), and the offset mapping returned with the token sequence gives each token's span. Let BSB_S and BTB_T be the byte offsets at which the source and receiver place boundaries. They always agree at the two ends of the context, and their common boundaries BS∩BTB_S\cap B_T cut the context into blocks, each containing at least one token of each model. For receiver row ii in block b(i)b(i), the source feature is the attention-mass-weighted pooling of the source rows in that block, concatenated over the selected source layers:

xi=concat⁡ℓ∈LS(∑j∈b(i)πj,[KS,jℓ;VS,jℓ]),πj∝source attention mass at j.\mathbf{x}_i=\operatorname{concat}_{\ell\in\mathcal{L}_S}\Big(\sum_{j\in b(i)}\pi_j\\,[\mathbf{K}^{\ell}_{S,j};\mathbf{V}^{\ell}_{S,j}]\Big),\qquad \pi_j\propto\text{source attention mass at }j .

On a block holding one token of each model, π\pi is a point mass and xi\mathbf{x}_i is exactly the source row, so the construction reduces to the within-family identity case.

Shared context representation. The context is shared as text: each model tokenizes the same string itself. Nothing in the pipeline requires the two models to share a tokenizer at serving time either, and computing the block map from the two offset mappings costs under 0.3 ms at 8K context, which is included in the latency numbers of Table 15.

Position coding. Positions never enter the pairing. Keys are mapped with rotary encoding stripped and re-rotated to the receiver's own positions at injection, so the correspondence is between content rows, not between position indices, and the two token sequences may differ in length with no realignment of position codes. This is the same property that lets a mapper fitted at one context length be applied at another. The receiver's cache always has exactly one row per receiver token, so NN, the position weights ww and uu, the probe index set and the injection hook are all unchanged from the within-family case.

Degenerate cases. Where tokenizers normalize whitespace or control characters differently, stretches can occur with no shared boundary. We align on the receiver's normalized byte string and record positions whose next common boundary lies more than 16 bytes away; these are under 0.2% of positions and are excluded from the second moments. They are never excluded from the served cache, which always covers every receiver position.

Rescoped future work. The conclusion now says "learning the cross-tokenizer correspondence rather than deriving it from byte boundaries", which is the genuinely open problem; Appendix B states the two limitations of the present construction (it needs the tokenizations to agree somewhere, which fails under sharply differing fertility, and on non-one-to-one blocks it assumes the pooled source rows carry what the receiver row needs). Appendix I also now records a negative result: we tried learning a soft alignment by optimal transport on hidden-state similarity, and on these pairs it changed CTI by 0.0120.012 on average while adding a per-request solve, so we kept the byte boundaries.

W2: The appendix uses the empirical quantile of the calibration score to construct the threshold, and then uses the calibration loss to select the control parameters ...

A2: This is the sharpest technical point in the reviews, and the reviewer has correctly identified the exact place where the argument could break. Theorem 1 of Angelopoulos et al. treats {Li(λ)}λ∈Λ\{L_i(\lambda)\}_{\lambda\in\Lambda} as a fixed family of functions; if the quantiles τj\tau_j that define the rungs were estimated on the same requests whose empirical risk is then controlled, the loss curves would depend on that sample and the exchangeability argument would not apply as stated. Monotonicity alone does not repair this.

We use an independent partition, and we should have said so. The revision makes it explicit at three levels.

  1. In the method (Section 4.3): the rung boundaries are estimated, so we keep that estimate apart from the calibration that certifies it. One set of requests fixes the score quantiles; a second, disjoint set supplies the conformal calibration.
AuthorsOpenReview ↗

Response to Reviewer NJFU (Part 2)

(Following above Part 1)

  1. As a proposition (new Proposition 7, Appendix A.7). Let Dq\mathcal{D}_{\mathrm{q}} and Dcal\mathcal{D}_{\mathrm{cal}} be disjoint, with Dcal∪{test}\mathcal{D}_{\mathrm{cal}}\cup\{\text{test}\} exchangeable conditionally on Dq\mathcal{D}_{\mathrm{q}}. Build τ0,τ1,τ2\tau_0,\tau_1,\tau_2 from Dq\mathcal{D}_{\mathrm{q}} alone and select λ^\widehat{\lambda} by Eq. (3) on Dcal\mathcal{D}_{\mathrm{cal}} using the monotonized loss. Then E[Ln+1(λ^)∣Dq]≤α\mathbb{E}\big[L_{n+1}(\widehat{\lambda})\mid\mathcal{D}_{\mathrm{q}}\big]\le\alpha almost surely, and hence E[Ln+1(λ^)]≤α\mathbb{E}\big[L_{n+1}(\widehat{\lambda})\big]\le\alpha. The proof is short: fixing Dq\mathcal{D}_{\mathrm{q}} makes λ↦τj(λ)\lambda\mapsto\tau_j(\lambda) deterministic, so the rung map is a fixed family of decision rules and each L~i(λ)\widetilde{L}_i(\lambda) is a fixed measurable function of request ii alone, bounded by 11, non-increasing and right-continuous in λ\lambda with L~i(1)=0\widetilde{L}_i(1)=0. Theorem 1 then applies conditionally, and the tower property gives the marginal statement. We also added a short paragraph before the proposition stating explicitly why the split is needed, so a reader does not have to reconstruct the reviewer's objection to see the point.

  2. In the protocol (Appendix H). We draw two disjoint sets of 300 requests per workload, both disjoint from the test split. Only Dcal\mathcal{D}_{\mathrm{cal}} needs dense-prefill references, so the split costs much less than doubling the calibration budget: 1.3 GPU-minutes for Dq\mathcal{D}_{\mathrm{q}} against 9.2 for Dcal\mathcal{D}_{\mathrm{cal}}, and we updated the per-pair calibration figure from 9.2 to 10.5 GPU-minutes accordingly (Table 14).

For completeness we also report what reuse would cost: pooling the two sets moves the selected thresholds by at most one grid step, work saved at α=0.10\alpha=0.10 by 0.006, and empirical risk by 0.003. We report the split version anyway, since only it is covered by Proposition 7.

W3: The 18 model pairs share multiple source models and receiving models ...

A3: We agree with the diagnosis and, on checking, the reviewer's suspicion is partly correct: some of the pooled 0.91 does come from the separation between regimes. The new Appendix F reports this honestly and adds the held-out tests the reviewer asks for.

One clarification first. CTI has no parameters fitted across pairs; it is a deterministic functional of two models and a calibration corpus. So holding out a model cannot leak in the way it would for a trained predictor, and the substantive issue is stratification rather than leakage. We now say this explicitly.

Stratified and partial correlations with intervals (new Table 10), percentile bootstrap over 10⁴ resamples of pairs:

Evaluation setnnCTIR2R^{2}
All directed pairs180.91 [0.78, 0.97]0.34 [−0.14, 0.69]
Within-family only120.73 [0.31, 0.92]0.39 [−0.20, 0.78]
Cross-family only60.83 [0.37, 0.99]0.20 [−0.66, 0.89]
Cross-checkpoint only41.00 [0.40, 1.00]n/a
Family distinction partialled out180.76 [0.46, 0.91]0.34 [−0.09, 0.66]
Held-out families, all80.95 [0.76, 1.00]0.43 [−0.26, 0.86]
Held-out families, cross-family60.89 [0.43, 1.00]0.26 [−0.60, 0.91]

So the pooled gap of about 0.57 between CTI and R2R^{2} becomes a gap of about 0.42 once the regime effect is removed. It shrinks, as the reviewer anticipated, but it does not disappear, and the within-family value of 0.73 is significant under a permutation test (p=0.007p=0.007) while the 0.39 of R2R^{2} is not (p=0.22p=0.22). These stratified numbers are now also previewed briefly in Section 5.2, since we think a reader deserves them in the main text rather than only in an appendix.

Reserved-family tests. Removing all pairs involving a family and scoring the index on exactly those removed pairs gives 0.97 over the nine pairs involving Qwen3, 0.90 over the eight involving Llama-3.x, and 0.89 over the seven involving Gemma-3. We also did this by receiver, but the resulting groups have two to four pairs each and the statistic is uninformative there (median 1.00, minimum 0.50 on a group of three); we report it for completeness and draw no conclusion from it. The small-group problem is precisely why we added new families instead.

Two entirely new families (new Table 11). Mistral and Phi-4 were held out while the analysis, the hyperparameters and the probe design were fixed, and were run afterwards with the protocol unchanged. They appear in no main-text number and in none of the figures over the 22 configurations. They supply eight directed pairs:

AuthorsOpenReview ↗

Response to Reviewer NJFU (Part 3)

(Following above Part 2)

PairCTIRetentionR2R^{2}CacheCert workCacheCert risk
Mistral 7B → Mistral-Small 24B0.760.90 ± 0.0280.710.670.082
Phi-4-mini → Phi-40.690.84 ± 0.0340.640.630.085
Llama 8B → Mistral 7B0.570.69 ± 0.0460.620.450.088
Qwen3 8B → Mistral 7B0.520.72 ± 0.0440.690.410.090
Mistral 7B → Qwen3 8B0.450.64 ± 0.0510.580.330.092
Phi-4-mini → Qwen3 8B0.310.54 ± 0.0560.710.260.093
Mistral 7B → Phi-40.220.57 ± 0.0600.550.140.095
Gemma 4B → Phi-4-mini †0.110.45 ± 0.0650.610.050.097

CTI reaches 0.95 Spearman over the eight and 0.89 over the six that cross families, against 0.43 and 0.26 for R2R^{2}, with nothing refitted. The absolute level transfers too, not only the ranking: the four pairs scoring above 0.45 all retain above 0.64, and the single pair below 0.15 falls under the threshold at which we call a backbone failed, where CacheCert again absorbs the problem as cost rather than as risk. Phi-4-mini → Qwen3 8B is a clean illustration of the paper's central claim on genuinely unseen models: its R2R^{2} of 0.71 is the second highest in the table while its retention is the second lowest, and the index places it correctly.

Q1: How do six cross family experiments deal with the differences in token boundaries and sequence lengths generated by different word breakers? How are cache rows, probe locations, and paired covariance aligned?

A4: A1 above gives the construction; here we answer the three specific sub-questions directly, and Appendix B now contains all of this.

  • Token boundaries and sequence lengths. Boundaries are handled by taking the intersection of the two boundary sets and working at block granularity. Sequence lengths are allowed to differ and are never reconciled: the receiver cache has one row per receiver token throughout, and the source side is pooled into that indexing. The measured length ratios run from 0.90 to 1.11 across the six pairs.
  • Cache rows. Paired as described: receiver row ii maps to the attention-mass-weighted pooling of the source rows in block b(i)b(i). The share of receiver rows sitting in one-to-one blocks is 0.86 for the Qwen3/Llama pairs, 0.74 for Qwen3/Gemma and 0.71 for Llama/Gemma; these blocks carry an even larger share of the position weight ww (0.88 to 0.89, 0.78 to 0.79, 0.75 to 0.76), because positions that attract attention tend to be whole words that both tokenizers delimit.
  • Probe locations. Probe positions are sampled in receiver index space, as in the within-family case. The stratification over source attention mass is pushed forward through the block map: a block's source mass is distributed over its receiver rows in proportion to w.
  • Paired covariance. Σxx\Sigma_{xx}, Σyx\Sigma_{yx} and Σyy\Sigma_{yy} are accumulated over the paired rows above, with the same position weights, so Proposition 4 and Definition 1 apply unchanged.

How much of the result depends on this choice (new Table 5). Restricting the second-moment accumulation to one-to-one blocks changes CTI by at most 0.02 on any cross-family pair (mean 0.015) and leaves the ordering of the six untouched. The pooling rule matters a little more but still not much: an unweighted mean changes CTI by 0.014 on average and retention by 0.006, while taking the first token of a block instead of pooling changes CTI by 0.038 and costs 0.021 retention, which is why we pool. The pairing is therefore not what separates cross-family from within-family pairs; Appendix G identifies what does.

Q2: Are independent samples used to construct empirical quantile thresholds and perform conformal risk calibration? If the same batch of samples is reused, which theoretical result guarantees that the data dependence loss family still satisfies equation (12)?

A5: Yes, independent samples are used: two disjoint sets of 300 requests per workload, both disjoint from the test split, as now stated in Section 4.3 and Appendix H. The guarantee is new Proposition 7, which obtains Eq. (12) by conditioning on the quantile-estimation set and then applying the tower property; the full statement and proof are in A2 above and in Appendix A.7.

To answer the conditional in the reviewer's question directly: if the same batch were reused, we are not aware of a result that would license Eq. (12) as stated, and we do not claim one. The reusing variant would need either a Learn-then-Test style argument with a correction over the random threshold family, or a full-conformal treatment, and we did not pursue either. We report the empirical cost of the split so a reader can see what the rigour provides and what it costs (thresholds within one grid step, 0.006 of work saved, 0.003 of risk).

Reviewer_3g3POpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of reducing inference resources by understanding when/if a model's context can be transferred to a different model effectively. To this end, the paper introduces the cache transferability index to measure the suitability of a target model for behaving similarly to a source model. The CTI is found via a calibration corpus. The paper also introduces CacheCert which addresses transferability for a particular request.

The authors apply CTI and CacheCert to a sufficient amount of within- and across- family pairs, a sufficiently diverse set of datasets, and compare to a sufficient amount of relevant baselines. They find that, on average, CTI has a higher pair-level prediction quality and is relatively cheap.

Critical strengths

The problem the paper addresses is real and the proposed methods have potential for high impact. The empirical results are strong -- the paper includes a satisfactory amount of model pairs, datasets, and experimental settings.

Critical weaknesses

The writing has pretty strong tells of heavy AI use. This is not a deal breaker, but the paper as a whole would benefit from a more careful editing.

Given that the CTI depends on a calibration set C, it is confusing that C does not show up in its definition.

The Cost calculation in Section 4.2 does not seem complete to me. In particular, CTI requires |C| sequences to be passed through both models. This incurs a non-trivial cost.

Methods wise, the ablations study reveals the importance of the sensitivity weighting. A deeper sensitivity analysis related to the weighting is necessary.

Questions for authors

Please address the weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3g3P (Part 1)

W1: The writing has pretty strong tells of heavy AI use. This is not a deal breaker, but the paper as a whole would benefit from a more careful editing.

A1: Thank you, and we take the point. As disclosed in the AI Use Statement, an LLM assistant was used to copy-edit author-written drafts; we did not realise how strongly its characteristic habits had survived into the final text, and on re-reading we agree they had.

We have made a line-level editing pass over the whole paper. The most conspicuous tell was a repeated rhetorical template: the paper announced a tricolon and then enumerated it four separate times, with "Three properties of Eq. (4) guide the rest of the paper", "Three remarks clarify the scope and use of CTI", "Three points clarify the scope of this guarantee", and "Three clarifications have proved necessary in discussion", each followed by First/Second/Third. We have dissolved the First/Second/Third scaffolding into ordinary prose and varied the openers, so the pattern no longer repeats across the paper. We also removed a set of self-congratulatory or redundant connective sentences, for example "This makes the approximation acceptable, whereas it would not be if we derived a bound from the score" (which merely restated the preceding paragraph), merged sentence pairs that had been split purely for rhythm, and fixed the garbled sentence in Section 5.1 ("The other six cross model families and represent..."), now "The remaining six pairs cross model families and represent...".

The pass was deliberately subtractive: the revised main text is shorter than the original everywhere except where a reviewer asked for new material, which is also how we made room for the additions requested below without exceeding the page limit. We are of course happy to keep editing, and would welcome pointers to any specific passages that still read badly.

W2: Given that the CTI depends on a calibration set C, it is confusing that C does not show up in its definition.

A2: Agreed, and this was sloppy on our part, because the dependence is genuine and runs through every ingredient, not just one. We have made it explicit in three places.

  1. Notation. Definition 1 now reads CTI(S→T;C)\mathrm{CTI}(S \to T; \mathcal{C}), and the statement of the definition says that every second moment, sensitivity operator and amplification weight in the expression is estimated on C\mathcal{C}. Section 4.1 correspondingly introduces Q\mathcal{Q} as the query distribution induced by a calibration corpus C\mathcal{C} rather than leaving the corpus implicit.

  2. A remark on what that means. The list of properties following Definition 1 now closes with the observation that the index is a functional of C\mathcal{C} through all of its ingredients, so it predicts for the regime that corpus represents. We state that we draw C\mathcal{C} from a domain disjoint from every evaluation workload, precisely so the index is not tuned to the test distribution, and we point to the new measurement below.

  3. Measuring the dependence rather than asserting it away. Because the reviewer is right that C\mathcal{C} is a real argument of the index, we recomputed every score on three further corpora, each matched to the default in sequence count and length. The new Table 13 reports the result:

    Corpusmean abs. change in CTImax abs. change in CTISpearman vs default rankingSpearman vs retention
    FineWeb-Edu (default)——1.000.91
    C40.0180.0410.990.90
    The Stack (Python)0.0410.0940.960.86
    mC4 (five languages)0.0340.0770.970.88
    Equal mixture of the four0.0210.0480.990.91

    The level of the index is corpus-dependent and the ranking it induces is largely not, which is the behaviour screening actually needs. The code corpus is the clearest case: attention there is more peaked, so the key path contributes less and scores shift most, yet the ranking is still 0.96 against the default and still predicts retention at 0.86. We now recommend drawing C\mathcal{C} from the deployment's own traffic where it is available, and a mixture otherwise.

We also note that this is separate from, and complementary to, the calibration-size sweep that was already in the appendix.

W3: The Cost calculation in Section 4.2 does not seem complete to me. In particular, CTI requires |C| sequences to be passed through both models. This incurs a non-trivial cost.

A3: The reviewer is right that this point was indeed not explicitly stated, and we want to be clear that it was an exposition failure rather than an accounting one: the paired forward passes were always inside the 2.1 GPU-minutes we report, and they are in fact the dominant term. The text simply did not say so. We have fixed this in three ways.

AuthorsOpenReview ↗

Response to Reviewer 3g3P (Part 2)

(Following above Part 1)

  1. Table 14 is now fully itemized. The offline figure breaks down as:

    ComponentCost
    CTI source forward pass (256 × 1024 tok.)0.6 GPU-min
    CTI receiver forward pass (256 × 1024 tok.)0.9 GPU-min
    CTI second-moment accumulation, O(d2)O(d^{2}) per head0.1 GPU-min
    CTI propagation probe (64 seq., once per receiver)0.5 GPU-min
    CTI total, new pair2.1 GPU-min
    CTI total, new source for a known receiver0.7 GPU-min
    Fitted-mapper baseline (paired passes 1.6, fit 3.8, eval 79.2)84.6 GPU-min
  2. The comparison is like-for-like. The decomposition also makes clear that the fitted-mapper baseline pays the same 1.6 GPU-minutes of paired passes before it begins to fit, and then adds a ridge solve and a downstream evaluation on top. So the roughly fortyfold gap is between two complete procedures, not between a partial cost and a full one. We say this explicitly in Section 4.3 and again in the caption of Table 1, which now notes that the quoted cost includes the forward passes through both models.

  3. What amortizes, and what does not. Two of the four terms amortize, and this matters for a deployment with several candidate senders: the propagation probe depends only on the receiver, and the receiver-side quantities G,W,U\mathbf{G},\mathbf{W},\mathbf{U} do not depend on the source. If the receiver's calibration activations are retained (about 9 GB for a 32B receiver at this corpus size), a further source costs only the source pass and the cross-covariance accumulation, 0.7 GPU-minutes. We have added this row to Table 14 and a short paragraph in Appendix H explaining it.

We have also completed the online accounting in the same spirit. The source-side attention aggregation used to place the probes was previously not listed; it is 0.2% of prefill FLOPs, so the online total is 1.6% rather than the 1.4% we had quoted. We corrected that figure. Finally, since cost in time is what a deployer feels, Appendix H now reports measured end-to-end TTFT with every overhead on the serving path (see A7 to Reviewer NJFU).

W4: Methods wise, the ablations study reveals the importance of the sensitivity weighting. A deeper sensitivity analysis related to the weighting is necessary.

A4: We agree, and this was the single largest gap: the original ablation only toggled G between the identity and the full operator, which shows that the weighting matters but not how or why. The new Appendix D studies it on five axes. The main text now points there in Section 5.4; we kept the main-text addition to a single sentence so the detail stays in the appendix.

  1. Continuous interpolation. Define Gβ∝(1−β) tr⁡Gd I+β G\mathbf{G}_{\beta}\propto(1-\beta)\,\tfrac{\operatorname{tr}\mathbf{G}}{d}\,\mathbf{I}+\beta\,\mathbf{G}, normalized to fixed trace so only anisotropy varies. Ranking quality rises smoothly and without a threshold, which is what one expects if the operator corrects a systematic mismatch rather than fitting noise:

    β\beta00.250.500.751
    All pairs0.580.690.790.870.91
    Cross-family0.310.470.630.760.84

    The increments are consistently larger on cross-family pairs.

  2. How many directions are actually needed. Truncating G\mathbf{G} to its leading kk eigendirections (trace preserved), out of a per-head dimension of 128:

    kk12481632
    All pairs0.440.560.680.790.870.90
    Cross-family0.220.350.510.660.780.83

    Sixteen directions recover most of the effect and thirty-two come within a point of the full operator, which is consistent with the effective ranks reported in Table 6 and suggests a cheaper truncated implementation is viable.

  3. Which ingredient carries the effect. Dropping the attention reweighting κ\kappa while keeping the query second moment costs less (0.91→0.840.91\to0.84) than dropping the query second moment while keeping κ\kappa as a per-head scalar (0.91→0.670.91\to0.67). The dominant ingredient is therefore the direction of the receiver's queries, not the overall level of attention sensitivity. Of the two position weights, ww matters much more (0.83) than uu (0.89), which is the key-path/value-path asymmetry that Section 4.1 predicts.

  4. Stability of the estimate. G\mathbf{G} is itself an expectation over queries. Estimating it from {64,256,1024,4096}\{64,256,1024,4096\} queries per head against a 16,38416{,}384-query reference gives maximum CTI deviations of 0.042, 0.013, 0.004 and 0.001, with induced Spearman 0.86, 0.90, 0.91, 0.91. We use 1024.

AuthorsOpenReview ↗

Response to Reviewer 3g3P (Part 3)

(Following above Part 2)

  1. Why it matters more across families. We define the exposure of a pair as the share of its irreducible residual energy falling in the leading sixteen eigendirections of G\mathbf{G}, divided by what an isotropic residual would place there. The median is 1.3 within families, 3.2 across families, and 4.7 on the three pairs whose backbone fails. An unweighted statistic is blind to exactly this factor, which explains why its ranking degrades most in the regime where the factor varies most. This statistic also does the explanatory work in the new Appendix G.

  2. The same weighting online. Sweeping β\beta in the online score rather than in the index moves work saved at α=0.10\alpha=0.10 from 0.47 (β=0\beta=0) through 0.57 to 0.66 (β=1\beta=1), while empirical risk stays within 0.003 of 0.083. The weighting improves efficiency online exactly as it improves ranking quality offline, and in neither case does it affect validity.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers for the care they put into these reviews. Several comments pointed out gaps in content, and we have run new experiments to close them. All changes are highlighted in blue in the revised PDF. The main additions are:

  • Figure 1, a high-level overview of how the sensitivity operator, CTI, the online probe and the fallback ladder relate (LJbQ W1).
  • Appendix B, a full specification of the cross-tokenizer row pairing with per-pair statistics and an ablation (NJFU W1/Q1, LJbQ Q1).
  • Appendix D, a dedicated study of the sensitivity weighting: continuous interpolation, rank truncation, ingredient decomposition, estimation stability, and an exposure statistic (3g3P W4).
  • Appendix F, stratified and partial correlations with bootstrap intervals, leave-one-family-out results, and eight new directed pairs from two entirely held-out families (Mistral, Phi-4) (NJFU W3/Q3).
  • Appendix G, a decomposition of what drives failure on the three worst pairs (LJbQ W2/Q1).
  • Proposition 7 and an explicit two-subset calibration protocol (NJFU W2/Q2, LJbQ Q3).
  • Table 4 (validity regime of the linearization, LJbQ Q2), Table 13 (calibration-corpus dependence, 3g3P W2), Table 14 (fully itemized cost, 3g3P W3), Table 15 (end-to-end TTFT by context length, NJFU Q4).

The main text grows by one page, to the ten-page limit; the detail sits in the appendix.

Area_Chair_sFPfOpenReview ↗

Reviewers: please respond to the author rebuttal

Dear reviewers,

Thank you to those who have already engaged with the authors' responses. The author-reviewer discussion closes on November 18 (AoE).

If you have not yet replied, please read the authors' response to your review and post a reply here before then: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Engaging with the other reviews and the ongoing discussion is also welcome.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.