acceptodds
Under review as a conference paper at ICLR 2027

KV Thickets: Task Experts from Random Prefix-Cache Perturbations

Abstract

Neural Thickets reveal task-improving experts through random weight perturba- tions, but each candidate defines a distinct parameter state. In-context learning provides a natural starting point: a prefix steers a frozen model, and its key–value (KV) states permit continuous changes without editing discrete tokens. Can mod- est populations of small random prefix-KV perturbations also contain useful task experts? We introduce KV Thickets, which independently sample perturbations from prespecified distributions, select candidates on labeled examples, and reuse the selected perturbations across inputs while keeping both model weights and prefix text unchanged. Our vLLM implementation batches their evaluation under shared weights. Across seven models and seven benchmarks, 32 candidates yield a selection-split improver in 45 of 49 settings; at 256 candidates, coverage reaches all 49, with a mean improving fraction of 28.7%. On examples disjoint from selection, the top selection-ranked candidate and an eight-candidate plurality vote improve mean task score by 2.70 and 4.83 points, respectively, over the unperturbed greedy baseline. These comparisons are descriptive because the evaluation includes previously examined benchmark pools. Weight-space experts provide a comple- mentary reference: their induced cache changes have mixed direct effects, but their blockwise magnitudes define another useful random-search profile. Together, these findings identify fixed-prefix KV states as a search space for reusable task adaptation under a single frozen model.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
KV cache, random perturbations, expert selection, model ensembling, language-model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kk9YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces KV Thickets, which transfers the random perturbation, task-based selection, and voting procedure of Neural Thickets to the KV states of a fixed prefix while keeping model weights unchanged. Experiments report improvements in most evaluated settings.

Critical strengths

The method provides a simple, gradient-free approach to task adaptation. Its shared-weight implementation allows multiple candidates to be evaluated using a single frozen model.

Critical weaknesses

The method’s potential to raise the model’s capability ceiling is unclear. Methods such as Neural Thickets, PPO, and GRPO modify model parameters to improve performance, whereas this method applies different perturbations to the same frozen model for different tasks. In a sense, each new task requires starting over, without accumulating improvements in the underlying model. Moreover, storing additional task-specific perturbations increases GPU memory requirements, which may substantially limit the method’s practical applicability.

The reported improvements are not fully convincing. The main results compare voting over eight selected candidates against a single greedy baseline. Random perturbations can themselves produce better or worse outputs, much as stochastic sampling does in GRPO. Appropriate baselines should therefore include eight-sample temperature-based self-consistency and voting over eight random KV candidates without task-based selection. Furthermore, in Conclusion and lines 24-25,the authors acknowledge previously examining some evaluation pools, raising concerns about possible test-set leakage. Evaluation on a completely new final test set would strengthen the evidence.

Questions for authors

Can the method support sustained capability improvement beyond selecting separate perturbations for individual tasks? How does storing task-specific perturbations affect GPU memory usage and scalability as the number of tasks increases?

How does the method compare with eight-sample self-consistency and voting over eight unselected random KV candidates under matched inference budgets? Can the authors confirm the reported gains on a previously unexamined final test set?

3 comments

Thank you for the feedback. We clarify the model interpretation and storage advantages below, followed by the revised experimental evidence.


Response to Critical Weaknesses

Capability and model interpretation. Once a fixed prompt has been prefilled into KV states, these states can be viewed as part of the effective model configuration that determines subsequent predictions.

In this functional sense, an expert is specified by both its weights and its fixed prefix state. Schematically, the two interventions can be written as (informal but may aid understanding):

Weight-space perturbation:

fW(⋅)=f(⋅,∣,θ+ϵ,;p∗)f^W(\cdot) = f(\cdot \\, | \\, \theta + \epsilon, \\; p^{\ast})

Prefix-KV perturbation:

fKV(⋅)=f(⋅,∣,θ,;p∗+ϵ)f^{KV}(\cdot) = f(\cdot \\, | \\, \theta, \\; p^{\ast} + \epsilon)

Here θ\theta denotes model weights, p∗p^{\ast} denotes the fixed prefix-context KV states, and ϵ\epsilon denotes a perturbation in the respective space.

Following a conventional LLM written as f0(⋅,∣,x<t)=f(⋅,∣,θ,;p∗,x<t)f^0(\cdot \\, | \\, x_{ < t}) = f(\cdot \\, | \\, \theta, \\; p^{\ast}, x_{ < t}) , with each configuration fixed, the same input x<tx_{ < t} gives three different "LLMs" (again, an informal description):

f0(⋅,∣,x<t),;fW(⋅,∣,x<t) and fKV(⋅,∣,x<t)f^0(\cdot \\, | \\, x_{ < t}), \\; f^W(\cdot \\, | \\, x_{ < t}) \text{ and } f^{KV}(\cdot \\, | \\, x_{ < t}) So, in a sense, a chosen prefix context integrated into a given LLM can be seen as changing the underlying model.

In another sense, a selected prefix-KV perturbation therefore defines a reusable predictor and selecting an expert for a new task does not discard previously selected experts: they remain available under the same base model. This expands the collection of task-specific predictors, although it does not explicitly establish continual learning in the base weights or cross-task transfer.

Storage and practical use. Compact prefix-KV perturbations make an expert library practical without keeping every expert on the GPU. In our geometry audit, a dense BF16 prefix-KV perturbation occupies 6.0–133.5 MiB; corresponding weight perturbations require 62.5–328.3 times as much storage across the seven audited 0.5–8B models (Section 3.5). KV states can be archived in RAM or on SSD and loaded as needed. Their smaller payload reduces the data transferred per expert switch, while our vLLM implementation batches different experts under one resident model. This offers a basis for industrial LMaaS and potentially storage-constrained edge deployments. Weight-space experts can also be offloaded or regenerated; the KV advantage is the smaller materialized state, faster speed in transferring and shared-weight execution.

Revised evidence. In response to these concerns, we have rerun the experiments to compare selected and score-blind random-candidate voting at the same K=4 inference budget, alongside a matched comparison with weight-space search across 46 model–task settings.

Our central finding is higher mean selection-split expert density in KV space than in weight space: 31.28% for KV versus 20.48% for Neural Thickets (NT) in our experiments. NT argues that task experts are sufficiently dense around contemporary trained models for random search to serve as an effective post-training approach. Our higher observed density suggests potentially more efficient random search: under the tested sampling protocols, a larger fraction of KV candidates improves on the selection baseline, so fewer candidate evaluations may be needed to find an improving expert. In measured runtime, KV also uses 24.54% less summed offline search time than NT under this protocol (Section 4.2; Table 1).


Response to Questions for Authors

Sustained improvement and memory. Please see Capability and model interpretation and Storage and practical use above. Selected experts remain reusable as the task library grows, without requiring every task-specific prefix-KV perturbation to reside on the GPU.

Matched-budget voting. The requested K=8 comparison has not yet been evaluated. Mean test-score gains over P0 (points), equally averaged over 46 model–task settings:

Search candidates (N)Vote size (K)KV selectedKV randomNT selectedNT random
1284+3.61+0.79+5.40+1.73

Previously unexamined test data. We reran the experiments on previously unexamined test rows, with candidate IDs fixed before evaluation.

Thank you for the response. Regarding W1, the authors provide only a theoretical explanation of how model behavior can change even when the weights remain frozen. However, they offer no evidence that improvements on one task can transfer to another or that the underlying model can achieve sustained, cumulative improvements in general capabilities. I therefore consider my concerns only partially addressed and will maintain my current rating.

Thank you for clarifying.

Our study concerns task-specific expert discovery rather than cumulative general-purpose learning.

This distinction also applies to the original Neural Thickets (NT) sampling–selection–voting procedure: NT Section 2.2 finds task specialists rather than uniformly stronger generalists, and modifying weights alone does not establish cross-task transfer.

Our contribution is to show that prefix-KV search achieves higher expert density with lower measured search cost and smaller expert payloads under our comparison protocol (or simpler, KV space is another useful space rich in task-specific experts against weight space). Cross-task transfer and cumulative capability improvement remain outside the evidence presented here.

We hope this clarification helps distinguish the broader goal of cumulative learning from the task-specific adaptation studied here, and that our contribution can be assessed within this scope.

Reviewer_Mqw5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces KV Thickets, a Neural Thickets method that searches for task-improving modules by randomly perturbing the key–value (KV) cache of a fixed textual prefix while keeping the model weights and prefix unchanged. Experiments across seven models and benchmarks imrpove over the unperturbed baseline.

Critical strengths

KV Thickets is a simple, training-free inference-time adaptation method that reuses KV-cache perturbations to construct task-specific experts without modifying model weights. Its empirical results show improvements over the unperturbed baseline.

Critical weaknesses

Marginal novelty. KV Thickets is essentially standard Neural Thickets applied to the KV cache of the prefix.

Unclear motivations. Why should peturbing KV states and selecting candidate from the perturbations have advanatges over standard Neural Thickets methods? If the concern is that prompt space is discrete, then a naive baseline would be to embed the prompt into a continuous space then conduct random sampling from that space. Have the authors compared KV Thickets to such a baseline?

Experiments. How does KV Thickets perform against standard Neural Thickets? The authors only show improvements over the unpeturbed baseline.

Additionally, the paper suffers from significant writting issues. I have a hard time trying to grasp the contributions of the paper and its position agaisnt existing literature. Here are some noticable issues.

Abstract. The abstract does not clearly set up the problem the paper is trying to address. It is unclear why the "distinct parameter state" should be an issue. The second sentence "In-context learning provides a natural starting point:..." does not state clearly what the "starting point" is for.

Introduction. The writing is difficult to follow for a reader not familiar with Neural Thickets literature. There should be a paragraph to set up the problem and introduce Neural Thickets properly before going into KV Thickets.

Questions for authors

N/A

1 comment

Thank you for the feedback. We apologise for the confusing wording and the lack of a self-contained explanation. We have revised the wording.

Novelty and Motivation. We empirically show that, beyond weight space, Key-Value prefix (also the input of a given LLM) provides a space where the task experts are dense (even a random noise would often lead the model to an expert for large LLMs), and there's the trend that the density grows as the model scales up, which is also observed in Neural Thickets.

The sampling–selection–voting procedure follows Neural Thickets. Our contributions are the empirical characterization of task-expert density in prefix-KV space and the shared-weight execution of multiple KV experts.


To help readers understand our approach, we provide an alternative, informal explanation.

We can express the two interventions as follows:

Weight Space Perturbations:

fW(⋅)=f(⋅,∣,θ+ϵ,;p∗)f^W(\cdot) = f(\cdot \\, | \\, \theta + \epsilon, \\; p^{\ast})

KV Space Perturbations:

fKV(⋅)=f(⋅,∣,θ,;p∗+ϵ)f^{KV}(\cdot) = f(\cdot \\, | \\, \theta, \\; p^{\ast} + \epsilon)

where θ\theta denotes the LLM parameters, p∗p^{\ast} denotes the prefix-context states injected into the LLM ff, and ϵ∼N(0,σ)\epsilon \sim N(0, \sigma) denotes the perturbations.

A conventional LLM can be written as f0(⋅,∣,x<t)=f(⋅,∣,θ,;p∗,x<t)f^0(\cdot \\, | \\, x_{ < t}) = f(\cdot \\, | \\, \theta, \\; p^{\ast}, x_{ < t}). With each configuration fixed, the same input x<tx_{ < t} gives three different "LLMs" (again, an informal description):

f^0(\cdot \\, | \\, x_{ < t}), \\; f^W(\cdot \\, | \\, x_{ < t}) \text{ and } f^{KV}(\cdot \\, | \\, x_{ < t}) \tag{*}

This motivates exploring KV states and formulating the problem this way.

Contribution. Our contributions are twofold:

  1. Observation. Neural Thickets (ICML 2026 Spotlight) claimed that task experts are dense around pretrained model weights; we find that, without perturbing the whole model, KV state space is even denser with task experts.

  2. Application. a. Our vLLM implementation provides a basis for language-model-as-a-service (LMaaS) systems serving multiple task experts. Each request carries its selected prefix-KV field, allowing requests for different experts to share a batch under one resident model while maintaining separate runtime KV states. Selected fields can be reused across inputs without switching model weights as the requested expert changes (Section 3.5). b. The same shared-weight execution also improves offline search efficiency: KV search uses 24.54% less summed search time than NT-style weight search under our matched protocol (Section 4.2).

Comparison with Neural Thickets. Across 46 matched model–task cells, KV and NT-style weight search achieve mean selection-split expert densities of 31.28% and 20.48%, respectively. Their selected four-candidate votes improve test scores over P0 by 3.61 and 5.40 points. KV uses 24.54% less summed offline search time under this protocol (Table 1; Figures 3–4).

Continuous prompt embeddings. We appreciate your suggestion. As discussed in Novelty and Motivation, we explore the neighborhood of the KV states and how perturbations affect the LLM's performance. Readers can view random Gaussian perturbations as probes of the latent space. Rather than defining a new space and sampling from it, we consider this approach simpler, more elegant, and consistent with Occam's razor.

There are multiple ways to search KV space, but our aim is to show empirically that task experts are even denser in KV space than in weight space.

Writing. Thank you again for identifying the writing issues directly. We are working to make the paper as readable as possible. We revised the abstract to state the research question explicitly and expanded the introduction to explain Neural Thickets before introducing KV Thickets. We welcome further suggestions and will address them promptly in the revision.

Reviewer_bz8TOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores reusable task experts generated by randomly perturbing the prefix KV‑cache of frozen large‑language models. It runs experiments across multiple models and tasks, and makes empirical observations about searching useful KV‑cache states. The method adopts a workflow consisting of random perturbation, task‑based candidate filtering, and ensemble voting over candidate outputs.

Critical strengths
  1. It raises a meaningful practical research question, investigating whether reusable task‑specialised behaviours can be uncovered through perturbations in the prefix KV‑cache space without updating model weights. It provides initial empirical exploration covering a variety of models and tasks, offering preliminary insights for cache‑oriented LLM adaptation research.
  2. The work stays within the frozen‑model paradigm. It operates on the KV‑cache as a lightweight intervention point, which has practical relevance for real‑world settings where model weight modification is not permitted.
Critical weaknesses
  1. Distinction from existing prior work is not adequately established. The overall pipeline logic shares core mechanics with the existing Neural Thickets framework, with the main change moving perturbations from weight space to prefix KV‑cache space. Meanwhile, existing literature has already studied context adaptation and KV‑cache manipulation techniques. Without fair controlled comparisons against these related approaches, the unique value brought by the proposed random KV‑cache search cannot be clearly identified, weakening the paper’s core contribution.
  2. Empirical results fail to disentangle gains from the KV‑cache candidate search and gains from ensemble aggregation. Performance improvements observed cannot be fully attributed to the quality of perturbed KV‑cache candidates, as ensemble voting itself contributes to final performance. Missing baseline comparisons prevent us from understanding the actual value of the search process. In addition, though the design supports weight‑shared batching, each candidate still needs separate generation runs. Without systematic resource overhead evaluation, the practical efficiency merits of the approach remain unvalidated.
  3. Statistical evidence supporting claims around task‑expert density is insufficient. Expert identification depends on performance measured over the selection partition, which cannot guarantee stable generalisation to unseen test data. Relevant analysis relies on nested candidate pools and only one sampled candidate collection for each model‑task setup, lacking independent repeated sampling. There is insufficient evidence to reliably back the core claim regarding reusable task experts existing within KV‑cache space.
Questions for authors
  1. Under matched task data and search budget settings, can you provide controlled comparisons against closely‑related prior methods, to clarify what unique advantages your random prefix KV‑cache perturbation method delivers?
  2. Can you separate and quantify the respective contributions from your KV‑cache search procedure and the ensemble‑voting aggregation step, using reasonable baseline comparisons under equal inference cost?
  3. Since each perturbed candidate requires its own generation trajectory, can you report practical runtime and resource overhead metrics relative to standard model inference?
  4. Can you address whether your observations on task‑expert density hold across multiple independently sampled candidate banks, and elaborate on the relation between performance seen on the selection partition and real‑world test‑set generalisation for these KV‑cache experts?
1 comment

Thank you for the feedback. We have rerun the experiments with matched weight-space and random-candidate controls, and report density, test performance, and search cost separately.


Response to Critical Weaknesses

Contribution and controlled comparison. The sampling–selection–voting procedure follows Neural Thickets (NT). Our contribution is the empirical study of expert density in the neighbourhood of a given prefix-KV state, by adding perturbations. We compare KV and NT across 46 model–task settings using matched data and candidate budgets. Mean expert density is 31.28% for KV versus 20.48% for NT; four-candidate votes improve mean test scores over P0 by 3.61 and 5.40 points, respectively (Section 4.1; Table 1). Thus, higher KV density and higher NT selected-vote accuracy are distinct findings.

Selection, voting, deployment and cost. Selection & voting. We compare selected and score-blind random-candidate voting at K=4 to measure the benefit of selection at the same voting budget (results below). Deployment. Our vLLM implementation batches different experts under one resident model, while each candidate retains its own generation trajectory and runtime KV state, providing a basis for language-model-as-a-service (LMaaS) systems serving multiple task experts for multiple different requests. Cost. Offline KV search takes 152.17 GPU-hours versus 201.64 for NT, a 24.54% reduction under this protocol (Sections 3.5 and 4.2).

Density and generalisation. P0 denotes the unperturbed baseline model's performance. Density is the fraction of the full candidate bank scoring strictly above P0 on selection rows. In the rerun, which includes larger models, mean KV density remains at 30.94%–31.30% across the four search budgets, versus 20.38%–20.69% for NT (Table 3). Selected experts also improve performance through voting on separate test examples from the same benchmark, without further selection or adjustment.


Response to Questions for Authors

Comparison with prior methods. We added matched NT comparison. It compares the two perturbation spaces within the same sampling–selection–voting framework, establishing differences in density, accuracy, and search cost.

Search versus aggregation. Expert density (%) and test-score gains over P0 (points), equally averaged over 46 model–task settings (Table 3).

Search candidates (N)Vote size (K)KV density (%)NT density (%)KV selectedKV randomNT selectedNT random
128431.2820.48+3.61+0.79+5.40+1.73

At the same K=4 voting budget, selection adds 2.82 points over random-KV voting.

Runtime and resources. Summed offline search time across 46 model–task settings (Table 1):

KV (GPU-hours)NT (GPU-hours)KV time reduction relative to NT
152.17201.6424.54%

Shared weights avoid candidate-specific weight loading, but a K-member vote still requires K generation trajectories. Prefix-KV perturbations are substantially lighter than NT weight perturbations: across seven audited 0.5–8B models, NT requires 62.5–328.3 times as much storage for dense BF16 perturbations (Section 3.5). This gives KV a clear per-expert payload advantage, reducing the data stored and transferred when saving or loading experts.

Independent sampling and test generalisation. Mean expert density across 46 model–task settings at each checkpoint (Table 3):

Search candidates (N)Vote size (K)KV density (%)NT density (%)
32131.2520.65
64230.9420.69
96331.3020.38
128431.2820.48

These stable densities support consistency across search budgets within the sampled banks, although the checkpoints are not independent replications. The test gains above support within-benchmark generalisation of selected-expert voting. Generalisation beyond these benchmarks is discussed as a limitation in Section 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.