- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces KV Thickets, which transfers the random perturbation, task-based selection, and voting procedure of Neural Thickets to the KV states of a fixed prefix while keeping model weights unchanged. Experiments report improvements in most evaluated settings.
Critical strengths
The method provides a simple, gradient-free approach to task adaptation. Its shared-weight implementation allows multiple candidates to be evaluated using a single frozen model.
Critical weaknesses
The method’s potential to raise the model’s capability ceiling is unclear. Methods such as Neural Thickets, PPO, and GRPO modify model parameters to improve performance, whereas this method applies different perturbations to the same frozen model for different tasks. In a sense, each new task requires starting over, without accumulating improvements in the underlying model. Moreover, storing additional task-specific perturbations increases GPU memory requirements, which may substantially limit the method’s practical applicability.
The reported improvements are not fully convincing. The main results compare voting over eight selected candidates against a single greedy baseline. Random perturbations can themselves produce better or worse outputs, much as stochastic sampling does in GRPO. Appropriate baselines should therefore include eight-sample temperature-based self-consistency and voting over eight random KV candidates without task-based selection. Furthermore, in Conclusion and lines 24-25,the authors acknowledge previously examining some evaluation pools, raising concerns about possible test-set leakage. Evaluation on a completely new final test set would strengthen the evidence.
Questions for authors
Can the method support sustained capability improvement beyond selecting separate perturbations for individual tasks? How does storing task-specific perturbations affect GPU memory usage and scalability as the number of tasks increases?
How does the method compare with eight-sample self-consistency and voting over eight unselected random KV candidates under matched inference budgets? Can the authors confirm the reported gains on a previously unexamined final test set?
3 comments
Thank you for the response. Regarding W1, the authors provide only a theoretical explanation of how model behavior can change even when the weights remain frozen. However, they offer no evidence that improvements on one task can transfer to another or that the underlying model can achieve sustained, cumulative improvements in general capabilities. I therefore consider my concerns only partially addressed and will maintain my current rating.
Thank you for clarifying.
Our study concerns task-specific expert discovery rather than cumulative general-purpose learning.
This distinction also applies to the original Neural Thickets (NT) sampling–selection–voting procedure: NT Section 2.2 finds task specialists rather than uniformly stronger generalists, and modifying weights alone does not establish cross-task transfer.
Our contribution is to show that prefix-KV search achieves higher expert density with lower measured search cost and smaller expert payloads under our comparison protocol (or simpler, KV space is another useful space rich in task-specific experts against weight space). Cross-task transfer and cumulative capability improvement remain outside the evidence presented here.
We hope this clarification helps distinguish the broader goal of cumulative learning from the task-specific adaptation studied here, and that our contribution can be assessed within this scope.
Thank you for the feedback. We clarify the model interpretation and storage advantages below, followed by the revised experimental evidence.
Response to Critical Weaknesses
Capability and model interpretation. Once a fixed prompt has been prefilled into KV states, these states can be viewed as part of the effective model configuration that determines subsequent predictions.
In this functional sense, an expert is specified by both its weights and its fixed prefix state. Schematically, the two interventions can be written as (informal but may aid understanding):
Weight-space perturbation:
fW(⋅)=f(⋅,∣,θ+ϵ,;p∗)
Prefix-KV perturbation:
fKV(⋅)=f(⋅,∣,θ,;p∗+ϵ)
Here θ denotes model weights, p∗ denotes the fixed prefix-context KV states, and ϵ denotes a perturbation in the respective space.
Following a conventional LLM written as f0(⋅,∣,x<t)=f(⋅,∣,θ,;p∗,x<t) , with each configuration fixed, the same input x<t gives three different "LLMs" (again, an informal description):
f0(⋅,∣,x<t),;fW(⋅,∣,x<t) and fKV(⋅,∣,x<t) So, in a sense, a chosen prefix context integrated into a given LLM can be seen as changing the underlying model.
In another sense, a selected prefix-KV perturbation therefore defines a reusable predictor and selecting an expert for a new task does not discard previously selected experts: they remain available under the same base model. This expands the collection of task-specific predictors, although it does not explicitly establish continual learning in the base weights or cross-task transfer.
Storage and practical use. Compact prefix-KV perturbations make an expert library practical without keeping every expert on the GPU. In our geometry audit, a dense BF16 prefix-KV perturbation occupies 6.0–133.5 MiB; corresponding weight perturbations require 62.5–328.3 times as much storage across the seven audited 0.5–8B models (Section 3.5). KV states can be archived in RAM or on SSD and loaded as needed. Their smaller payload reduces the data transferred per expert switch, while our vLLM implementation batches different experts under one resident model. This offers a basis for industrial LMaaS and potentially storage-constrained edge deployments. Weight-space experts can also be offloaded or regenerated; the KV advantage is the smaller materialized state, faster speed in transferring and shared-weight execution.
Revised evidence. In response to these concerns, we have rerun the experiments to compare selected and score-blind random-candidate voting at the same K=4 inference budget, alongside a matched comparison with weight-space search across 46 model–task settings.
Our central finding is higher mean selection-split expert density in KV space than in weight space: 31.28% for KV versus 20.48% for Neural Thickets (NT) in our experiments. NT argues that task experts are sufficiently dense around contemporary trained models for random search to serve as an effective post-training approach. Our higher observed density suggests potentially more efficient random search: under the tested sampling protocols, a larger fraction of KV candidates improves on the selection baseline, so fewer candidate evaluations may be needed to find an improving expert. In measured runtime, KV also uses 24.54% less summed offline search time than NT under this protocol (Section 4.2; Table 1).
Response to Questions for Authors
Sustained improvement and memory. Please see Capability and model interpretation and Storage and practical use above. Selected experts remain reusable as the task library grows, without requiring every task-specific prefix-KV perturbation to reside on the GPU.
Matched-budget voting. The requested K=8 comparison has not yet been evaluated. Mean test-score gains over P0 (points), equally averaged over 46 model–task settings:
Previously unexamined test data. We reran the experiments on previously unexamined test rows, with candidate IDs fixed before evaluation.