- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether adaptation primarily elicits capabilities already present in foundation models. It decomposes adaptation-induced changes into an input-independent mean component and an input-specific residual, then evaluates interventions that retain or remove the shared component. Motivated by these results, the authors train a shift adapter consisting of one constant vector per layer. Across 17 models and 17 tasks, this adapter matches the average accuracy of rank-8 LoRA with substantially fewer trainable parameters. Further experiments examine pretraining checkpoints, adaptation geometry, activation steering, cross-model similarity, and randomly discovered experts.
Critical strengths
-
Broad empirical coverage. The study spans multiple architecture families, model sizes, tasks, and behaviors, generally using three training seeds. And the shift adapter and rank-8 LoRA both achieve good performance.
-
Useful interventions beyond descriptive similarity. Retaining and removing the shared component provides stronger evidence than measuring activation alignment alone. Experiments with unconstrained updates, models before instruction tuning, and alternative probe estimates help assess the robustness of the observation.
-
Careful main task comparisons. Learning rates are selected separately for each method, model, and task using training-data validation splits. Paired comparisons and uncertainty estimates strengthen the central empirical findings.
-
Informative boundary cases. The arbitrary-mapping and payload experiments distinguish task operations from newly acquired content. The capacity-limit and early-pretraining results also reveal settings where constant shifts fall short of LoRA.
Critical weaknesses
-
The potentiality principle is stronger than the evidence establishes. Successful adaptation with a constant vector demonstrates an efficient parameterization, but does not establish that the resulting task expertise already existed before adaptation. The vector is learned from task data and can encode new information, as the arbitrary-table experiments explicitly demonstrate. Although the paper acknowledges input-dependent downstream effects and separates operations from payloads, it needs a more operational distinction between eliciting an existing capability and learning a new task through a frozen computational substrate.
-
The interventions support sufficiency more clearly than causal dominance. The shared component alone preserves 85% of LoRA’s task gain, but removing it still preserves 61%. More significantly, Appendix D.2 reports that position-matched removal preserves 85%, while projecting away the shared direction preserves 75%. These findings suggest substantial redundancy, interactions, or sensitivity to intervention design. They deserve greater prominence when interpreting the claim that the input-specific residual carries little adaptation gain. Energy concentration also does not directly establish functional importance.
-
The contribution relative to closely related work needs sharper positioning. Prior work already trains one steering vector per layer and reports competitive reasoning performance, while other work interprets LoRA adaptation as approximately constant steering. These connections are acknowledged, but the paper’s strongest contribution appears to be the breadth of its empirical investigation and intervention analysis. Direct comparisons with closely related trained-offset methods would better establish its additional contribution. See Sinii et al., Wang et al., and LoFiT.
-
Some auxiliary analyses reuse evaluation data for selection. Appendix G.2 selects and evaluates AxBench learning rates on the same held-out instructions. Appendix G.4 reports expert fractions at the noise scale maximizing that fraction on the held-out set. These choices can inflate the reported quantities and complicate interpretation of the correlations. The separate validation procedure used in the main task experiments should also be applied to these analyses.
Questions for authors
See Weakness.
4 comments
W3: The contribution relative to closely related work (Sinii et al., Wang et al., and LoFiT) needs sharper positioning, and direct comparisons with closely related trained-offset methods would better establish its additional contribution.
We thank the reviewer, and we made the distinction between these works and our paper clearer in the revision. The three works show that adaptation can be made parameter-efficient, while our paper shows for the first time that the expertise already exists in the foundation model, and thereby explains why parameter-efficient adaptation works.
The fundamental difference lies in what the works learn and show. Wang et al. (2025) [2] investigate out-of-context reasoning with a LoRA on the MLP of one layer of one model, and LoFiT (Yin et al., 2024) [3] trains offsets on selected attention heads. In both, the learned update is input-specific: a LoRA is applied to the input, and the offsets of LoFiT pass through the MLP of the block, so the output of the block changes differently for each input (new Appendix B.3). In contrast, the update our work learns is input-independent, motivating us to propose the potentiality principle. Sinii et al. (2025) [1] train one vector per layer with reinforcement learning for mathematical reasoning and see its input-independence as a limitation that caps the performance. All three works show that an update restricted to a few parameters adapts the model to their task, but none of them asks whether the capacity was already in the base model or when it formed, which is the focus of our work. Our work unifies these papers, as well as other adaptation papers, under a common principle that explains the adaptation behavior.
In that aspect, a direct comparison with these methods would only show that all of them are equally good on benchmarks and match LoRA, just like our approach does. However, our approach is even more restricted in terms of input independence, indicating that the capacity is already in the base model, which existing works do not show.
Overall, compared to these works, our contribution goes beyond yet another parameter-efficient method. Our work shows for the first time that the expertise for a task already exists in the foundation model: most of what LoRA and full fine-tuning add is the same for every input (for LoRA, 85% of its gain), a constant actualizes the expertise only once pretraining has formed the capacity (new Appendix F.11), and a readout of the base model estimates before training where a constant suffices (new Appendix F.12). The potentiality principle we propose also unifies existing work and sheds new light on adaptation as a whole: parameter-efficient fine-tuning, in-context learning, activation steering, model convergence, and neural thickets become different ways of reaching the actualizing states of a model. A trained shift adapter, for instance, reaches an adherence of 0.68 as a steering vector, against 0.17 for the best vector extracted from activations.
In response, we revised related work to make distinctions clearer and added a new table (Appendix A, Table R1) to present these differences.
W4: Appendices G.2 and G.4 reuse evaluation data for selection, and these choices can inflate the reported quantities.
We thank the reviewer for carefully reading the paper and finding that Appendices G.2 and G.4 reuse the evaluation data. We have revised the appendices to keep selection separate from evaluation in both, and the results remain consistent with minimal differences. In Appendix G.2, the rate of each concept is now chosen from the other 499 concepts. This selects the same rates as before on every concept, so the steering scores do not change. As a stricter check, we choose the rate on half of the concepts and score the other half, over 20 random splits: the shift adapter reaches 0.855 against 0.909 for LoRA, compared to 0.867 against 0.914 in the submission, so it keeps 94% of LoRA's score, against 95% before (table below). In Appendix G.4, the experts were already chosen on a selection split and confirmed on a disjoint held-out split. We now choose the noise size for each seed on the other seeds of the same model and task. The relation between experts and region size holds. Overall, our conclusions do not change, and we thank the reviewer for this valuable comment that helped to improve our work.
| Submitted | Revised | |
|---|---|---|
| G.2, steering score of the shift adapter, half split | 0.867 | 0.855 |
| G.2, steering score of LoRA, half split | 0.914 | 0.909 |
| G.2, paired difference, half split | -0.047 | -0.054 |
| G.4, expert fraction on GSM8K | 0.23 | 0.20 |
| G.4, AUC of region size for the pairs with an expert (72 pairs) | 0.82 | 0.82 |
| G.4, Spearman of region size and expert fraction (72 pairs) | +0.45 | +0.43 |
The main task comparisons already select rates on a validation slice of each training split, as the reviewer notes.
References
[1] V. Sinii, et al. Steering LLM Reasoning Through Bias-Only Adaptation. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.
[2] A. Wang, et al. Simple Mechanistic Explanations for Out-Of-Context Reasoning. arXiv preprint, 2025.
[3] F. Yin, et al. LoFiT: Localized Fine-tuning on LLM Representations. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Many thanks for the detailed response. I will keep the positive score.
We thank the reviewer for the careful reading and constructive feedback, for recognizing the broad empirical coverage, the careful main task comparisons, and that the interventions provide "stronger evidence than measuring activation alignment alone".
We thank the reviewer for the comment, and we agree that it is important to distinguish between eliciting an existing capability and learning a new task. However, the evidence can establish the distinction, and the experiments in our paper clearly point to the fact that the expertise existed before. We first summarize why the existing experiments in the paper support this, and then provide an additional operational distinction that could separate the two.
A capacity held in potentiality is one that the base model has but does not exercise, and actualizing it moves the model into a state in which it is exercised. A constant vector adds the same change to every input, while tasks such as performing a particular behavior or question answering need a different answer for each input. So when a constant vector gives the correct answer on inputs it has never seen, which is the case in our current experiments, the frozen base model must decide which answer each input gets. Across 17 tasks and 18 models, a shift adapter reaches a mean accuracy of 0.746, against 0.745 for a rank-8 LoRA. The two also solve the same held-out items: of the items one newly solves, the other newly solves 82%, against 51% by chance (revised Appendix D.2). This means that the capacity is already in the base model when a constant vector suffices on inputs never seen in training. Otherwise, the model could not generalize with the constant shift.
To provide an operational distinction between eliciting an existing capability and learning a new task, we design a new experiment along pretraining (new Appendix F.11). If a constant learned a new task through the frozen model, it would learn it at every checkpoint of pretraining, since the architecture and the parameter count stay the same. We take the pretraining checkpoints of OLMo-2-7B and adapt each to answer in 42 writing systems, that is, in languages written in their own non-Latin script, such as Thai. LoRA succeeds at both checkpoints: at 9B tokens, a rank-8 LoRA already produces 40 to 42 of them. A constant succeeds only at the later checkpoint: at 9B tokens, a shift adapter produces 7 to 8, and the shared component of the trained LoRA none, while at 105B tokens the shared component alone produces 37 to 40 of the 42 of the full LoRA. So the potentiality forms during pretraining, and only then does a constant elicit an existing capability rather than learn a new task.
In response to the reviewer's comment, Section 3 now states this operational distinction, and we added this new experiment in Appendix F.11.
We agree that LoRA still carries much of its gain in the input-specific component, and we rephrased this in the abstract, the introduction, and the conclusion: the shared component suffices for most of the gain and can be learned directly. We also agree that the interventions do not perfectly disentangle the two components. However, the key point from the intervention experiment is that a large gain lies in the shared component, which motivates us to propose the shift adapter, a learnable constant change, the same for every input. Trained directly, a shift adapter matches LoRA, 0.746 against 0.745 over 18 models and 17 tasks. That such an input-independent change exists and suffices is the main point of the paper and the basis of the potentiality principle, and it does not depend on whether the shared component dominates LoRA's update, which only serves as a motivation.