acceptodds
Under review as a conference paper at ICLR 2027

On Potentiality and Actuality Of Foundation Models

Abstract

Foundation models are built through large-scale pretraining, often followed by post-training, and are then adapted to specific tasks and behaviors. But, does the adaptation add truly new capabilities, or the base model already holds it? We decompose the update induced by different adaptations into a component shared across all inputs and a residual that varies with the input. We find that the input-dependent residual carries little of the adaptation gain: as evidence, a shift adapter that learns a constant vector per layer, identical for every input, reproduces the behavior of LoRA and full fine-tuning. We call this the potentiality principle of foundation models: the base model already contains the capacity needed for task-specific expertise. To adapt to a new task, the model just needs to be actualized, i.e., pushed in a certain direction via a constant, input-independent shift. We show the implications of this principle on parameter-efficient adaptation, steering, and convergence across models, and how it explains the neural thickets phenomenon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation models, parameter-efficient fine-tuning, model adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6mi2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether adaptation primarily elicits capabilities already present in foundation models. It decomposes adaptation-induced changes into an input-independent mean component and an input-specific residual, then evaluates interventions that retain or remove the shared component. Motivated by these results, the authors train a shift adapter consisting of one constant vector per layer. Across 17 models and 17 tasks, this adapter matches the average accuracy of rank-8 LoRA with substantially fewer trainable parameters. Further experiments examine pretraining checkpoints, adaptation geometry, activation steering, cross-model similarity, and randomly discovered experts.

Critical strengths
  1. Broad empirical coverage. The study spans multiple architecture families, model sizes, tasks, and behaviors, generally using three training seeds. And the shift adapter and rank-8 LoRA both achieve good performance.

  2. Useful interventions beyond descriptive similarity. Retaining and removing the shared component provides stronger evidence than measuring activation alignment alone. Experiments with unconstrained updates, models before instruction tuning, and alternative probe estimates help assess the robustness of the observation.

  3. Careful main task comparisons. Learning rates are selected separately for each method, model, and task using training-data validation splits. Paired comparisons and uncertainty estimates strengthen the central empirical findings.

  4. Informative boundary cases. The arbitrary-mapping and payload experiments distinguish task operations from newly acquired content. The capacity-limit and early-pretraining results also reveal settings where constant shifts fall short of LoRA.

Critical weaknesses
  1. The potentiality principle is stronger than the evidence establishes. Successful adaptation with a constant vector demonstrates an efficient parameterization, but does not establish that the resulting task expertise already existed before adaptation. The vector is learned from task data and can encode new information, as the arbitrary-table experiments explicitly demonstrate. Although the paper acknowledges input-dependent downstream effects and separates operations from payloads, it needs a more operational distinction between eliciting an existing capability and learning a new task through a frozen computational substrate.

  2. The interventions support sufficiency more clearly than causal dominance. The shared component alone preserves 85% of LoRA’s task gain, but removing it still preserves 61%. More significantly, Appendix D.2 reports that position-matched removal preserves 85%, while projecting away the shared direction preserves 75%. These findings suggest substantial redundancy, interactions, or sensitivity to intervention design. They deserve greater prominence when interpreting the claim that the input-specific residual carries little adaptation gain. Energy concentration also does not directly establish functional importance.

  3. The contribution relative to closely related work needs sharper positioning. Prior work already trains one steering vector per layer and reports competitive reasoning performance, while other work interprets LoRA adaptation as approximately constant steering. These connections are acknowledged, but the paper’s strongest contribution appears to be the breadth of its empirical investigation and intervention analysis. Direct comparisons with closely related trained-offset methods would better establish its additional contribution. See Sinii et al., Wang et al., and LoFiT.

  4. Some auxiliary analyses reuse evaluation data for selection. Appendix G.2 selects and evaluates AxBench learning rates on the same held-out instructions. Appendix G.4 reports expert fractions at the noise scale maximizing that fraction on the held-out set. These choices can inflate the reported quantities and complicate interpretation of the correlations. The separate validation procedure used in the main task experiments should also be applied to these analyses.

Questions for authors

See Weakness.

4 comments

We thank the reviewer for the careful reading and constructive feedback, for recognizing the broad empirical coverage, the careful main task comparisons, and that the interventions provide "stronger evidence than measuring activation alignment alone".

W1: The potentiality principle is stronger than the evidence establishes. Adaptation with a constant vector shows an efficient parameterization, not that the expertise existed before. The paper needs an operational distinction between eliciting an existing capability and learning a new task through a frozen computational substrate.

We thank the reviewer for the comment, and we agree that it is important to distinguish between eliciting an existing capability and learning a new task. However, the evidence can establish the distinction, and the experiments in our paper clearly point to the fact that the expertise existed before. We first summarize why the existing experiments in the paper support this, and then provide an additional operational distinction that could separate the two.

A capacity held in potentiality is one that the base model has but does not exercise, and actualizing it moves the model into a state in which it is exercised. A constant vector adds the same change to every input, while tasks such as performing a particular behavior or question answering need a different answer for each input. So when a constant vector gives the correct answer on inputs it has never seen, which is the case in our current experiments, the frozen base model must decide which answer each input gets. Across 17 tasks and 18 models, a shift adapter reaches a mean accuracy of 0.746, against 0.745 for a rank-8 LoRA. The two also solve the same held-out items: of the items one newly solves, the other newly solves 82%, against 51% by chance (revised Appendix D.2). This means that the capacity is already in the base model when a constant vector suffices on inputs never seen in training. Otherwise, the model could not generalize with the constant shift.

To provide an operational distinction between eliciting an existing capability and learning a new task, we design a new experiment along pretraining (new Appendix F.11). If a constant learned a new task through the frozen model, it would learn it at every checkpoint of pretraining, since the architecture and the parameter count stay the same. We take the pretraining checkpoints of OLMo-2-7B and adapt each to answer in 42 writing systems, that is, in languages written in their own non-Latin script, such as Thai. LoRA succeeds at both checkpoints: at 9B tokens, a rank-8 LoRA already produces 40 to 42 of them. A constant succeeds only at the later checkpoint: at 9B tokens, a shift adapter produces 7 to 8, and the shared component of the trained LoRA none, while at 105B tokens the shared component alone produces 37 to 40 of the 42 of the full LoRA. So the potentiality forms during pretraining, and only then does a constant elicit an existing capability rather than learn a new task.

In response to the reviewer's comment, Section 3 now states this operational distinction, and we added this new experiment in Appendix F.11.

W2: The interventions support sufficiency more clearly than causal dominance. The shared component alone preserves 85% of LoRA's task gain, but removing it still preserves 61%, and in Appendix D.2, 85% and 75%. These findings suggest substantial redundancy, interactions, or sensitivity to intervention design, and deserve greater prominence when interpreting the claim that the input-specific residual carries little adaptation gain.

We agree that LoRA still carries much of its gain in the input-specific component, and we rephrased this in the abstract, the introduction, and the conclusion: the shared component suffices for most of the gain and can be learned directly. We also agree that the interventions do not perfectly disentangle the two components. However, the key point from the intervention experiment is that a large gain lies in the shared component, which motivates us to propose the shift adapter, a learnable constant change, the same for every input. Trained directly, a shift adapter matches LoRA, 0.746 against 0.745 over 18 models and 17 tasks. That such an input-independent change exists and suffices is the main point of the paper and the basis of the potentiality principle, and it does not depend on whether the shared component dominates LoRA's update, which only serves as a motivation.

W3: The contribution relative to closely related work (Sinii et al., Wang et al., and LoFiT) needs sharper positioning, and direct comparisons with closely related trained-offset methods would better establish its additional contribution.

We thank the reviewer, and we made the distinction between these works and our paper clearer in the revision. The three works show that adaptation can be made parameter-efficient, while our paper shows for the first time that the expertise already exists in the foundation model, and thereby explains why parameter-efficient adaptation works.

The fundamental difference lies in what the works learn and show. Wang et al. (2025) [2] investigate out-of-context reasoning with a LoRA on the MLP of one layer of one model, and LoFiT (Yin et al., 2024) [3] trains offsets on selected attention heads. In both, the learned update is input-specific: a LoRA is applied to the input, and the offsets of LoFiT pass through the MLP of the block, so the output of the block changes differently for each input (new Appendix B.3). In contrast, the update our work learns is input-independent, motivating us to propose the potentiality principle. Sinii et al. (2025) [1] train one vector per layer with reinforcement learning for mathematical reasoning and see its input-independence as a limitation that caps the performance. All three works show that an update restricted to a few parameters adapts the model to their task, but none of them asks whether the capacity was already in the base model or when it formed, which is the focus of our work. Our work unifies these papers, as well as other adaptation papers, under a common principle that explains the adaptation behavior.

In that aspect, a direct comparison with these methods would only show that all of them are equally good on benchmarks and match LoRA, just like our approach does. However, our approach is even more restricted in terms of input independence, indicating that the capacity is already in the base model, which existing works do not show.

Overall, compared to these works, our contribution goes beyond yet another parameter-efficient method. Our work shows for the first time that the expertise for a task already exists in the foundation model: most of what LoRA and full fine-tuning add is the same for every input (for LoRA, 85% of its gain), a constant actualizes the expertise only once pretraining has formed the capacity (new Appendix F.11), and a readout of the base model estimates before training where a constant suffices (new Appendix F.12). The potentiality principle we propose also unifies existing work and sheds new light on adaptation as a whole: parameter-efficient fine-tuning, in-context learning, activation steering, model convergence, and neural thickets become different ways of reaching the actualizing states of a model. A trained shift adapter, for instance, reaches an adherence of 0.68 as a steering vector, against 0.17 for the best vector extracted from activations.

In response, we revised related work to make distinctions clearer and added a new table (Appendix A, Table R1) to present these differences.

W4: Appendices G.2 and G.4 reuse evaluation data for selection, and these choices can inflate the reported quantities.

We thank the reviewer for carefully reading the paper and finding that Appendices G.2 and G.4 reuse the evaluation data. We have revised the appendices to keep selection separate from evaluation in both, and the results remain consistent with minimal differences. In Appendix G.2, the rate of each concept is now chosen from the other 499 concepts. This selects the same rates as before on every concept, so the steering scores do not change. As a stricter check, we choose the rate on half of the concepts and score the other half, over 20 random splits: the shift adapter reaches 0.855 against 0.909 for LoRA, compared to 0.867 against 0.914 in the submission, so it keeps 94% of LoRA's score, against 95% before (table below). In Appendix G.4, the experts were already chosen on a selection split and confirmed on a disjoint held-out split. We now choose the noise size for each seed on the other seeds of the same model and task. The relation between experts and region size holds. Overall, our conclusions do not change, and we thank the reviewer for this valuable comment that helped to improve our work.

SubmittedRevised
G.2, steering score of the shift adapter, half split0.8670.855
G.2, steering score of LoRA, half split0.9140.909
G.2, paired difference, half split-0.047-0.054
G.4, expert fraction on GSM8K0.230.20
G.4, AUC of region size for the pairs with an expert (72 pairs)0.820.82
G.4, Spearman of region size and expert fraction (72 pairs)+0.45+0.43

The main task comparisons already select rates on a validation slice of each training split, as the reviewer notes.

References

[1] V. Sinii, et al. Steering LLM Reasoning Through Bias-Only Adaptation. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.

[2] A. Wang, et al. Simple Mechanistic Explanations for Out-Of-Context Reasoning. arXiv preprint, 2025.

[3] F. Yin, et al. LoFiT: Localized Fine-tuning on LLM Representations. Advances in Neural Information Processing Systems (NeurIPS), 2024.

Many thanks for the detailed response. I will keep the positive score.

Reviewer_FqyxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the nature of the changes learned during adaptation of an LLM after pretraining/postraining. The core result is that learning one constant vector per layer, which just yields an input-independent shift, can match LoRA and fine-tuning. This result suggests that LLMs already possess the requisite capabilities, and adaptation moves the network to a context where certain capabilities are emphasised. The paper connects this observation to a range of other findings in the literature.

Critical strengths

The empirical evaluations are extensive, covering 17 models and 17 tasks, and convincingly show that a fixed input-independent shift can move network behavior substantially. Moreover, this is in fact most of what LoRA adaptation is doing.

The connection to other related phenomena like neural thickets, platonic representation hypothesis, and activation steering is useful.

Overall the paper is well written.

Critical weaknesses

In my view the learned shift vector is another name for learning biases, and as noted in the introduction, prior work has shown that learning biases alone yields comparable performance (Ben Zaken et al., 2022 and Sinii et al., 2025). This appears to lessen the contribution of the paper. In particular, the core empirical result of the paper and referenced in the abstract, “a shift adapter that learns a constant vector per layer, identical for every input, reproduces the behavior of LoRA and full fine-tuning,” would appear to be largely contained in Ben Zaken, as well as the core conclusion (compare this paper’s conclusion “the base model already contains the capacity needed for task-specific expertise” to that from Ben Zaken’s abstract: “[results] support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.”) The paper could be improved by explaining the relationship between these works, including the specific differences between shift vector and bias training.

Questions for authors

What is the relationship between shift vectors and training biases?

What are the novel contributions of this paper relative to Ben Zaken et al 2022 and Sinii et al 2025?

3 comments

We thank the reviewer for recognizing that our evaluations "convincingly show that a fixed input-independent shift can move network behavior substantially" and that "this is in fact most of what LoRA adaptation is doing".

W1: The learned shift vector is another name for learning biases, and prior work has shown that learning biases alone yields comparable performance (Ben Zaken et al., 2022, and Sinii et al., 2025), so the core result and the core conclusion appear to be largely contained in Ben Zaken et al. (2022).

We respectfully disagree that the shift adapter is another name for bias training Ben Zaken et al. (2022) [1] and that our core result and conclusions are contained in prior works, for two reasons (with details in Q1 and Q2). First, the shift adapter and bias training are different parameterizations. Ben Zaken et al. (2022) train the biases together with a new classification layer for each task, while the shift adapter adds only vectors that are the same for every input and the model answers through its own output layer. Thus, the fundamental difference is that the update a shift adapter learns (i.e., one vector per layer) is added unchanged to the output of each block for every input, while the update bias training learns changes the output of each block differently for each input. This difference is crucial in proposing the potentiality and actuality principle that our paper rests on. To better formalize this relationship between shift adapter and bias training, we added a new Appendix B.3. On the other hand, Sinii et al. (2025) train one vector per layer with reinforcement learning for mathematical reasoning, but they see its input-independence as a limitation: "We hypothesize that this token-independence may cap performance" (their Appendix D). Our paper shows the opposite across a wide range of tasks: input-independence is not a limitation, since the expertise is already in the base model, and most of what LoRA itself adds is the same for every input (its shared component keeps 85% of its gain). In response to the Reviewer's comment, we expanded the Related work to make these distinctions clearer.

Second, our work goes beyond proposing yet another adaptation method and provides a unifying view of the adaptation principle. In that sense, our work is much broader than Ben Zaken et al. (2022), who study linguistic classification tasks on masked language models, and Sinii et al. (2025), on which our work also provides a new perspective, and we cited these in the related work of the submission in the original manuscript. In particular, our paper investigates why these efficient adaptations in Ben Zaken et al. and Sinii et al. works through the potentiality and actuality principle, thus helping the community to understand what adaptation changes in a foundation model, when potentiality arises during pretraining, what structure it has, and how it connects fields that have been studied apart, such as activation steering and neural thickets.

Q1: What is the relationship between shift vectors and training biases?

Shift adapters and bias training (Ben Zaken et al. (2022)) are similar in that both train a small set of vectors while keeping the model weights frozen. However, they are fundamentally different: the update a shift adapter learns is added unchanged to the output of each block for every input, while the update bias training learns changes this output differently for each input, through (i) the new classification layer and (ii) the biases themselves. We now added a formal view of this relationship in the new Appendix B.3, and we expand on these differences below.

(i) Ben Zaken et al. (2022) do not train the biases alone. They train the bias terms and the task-specific classification layer (their Section 3): a new layer, trained on the task, that reads the representation of the first token of each input and returns the label. In every result of their paper, the answer leaves the model only through this new layer. This trained layer adds an input-specific component. In contrast, in our work we completely remove the input-specific component of an update and test whether the adaptation gains remain. This lets us test whether the expertise is already in the base model, which is not possible to test in their framework, leading to the actuality and potentiality principle.

(ii) The shift adapter adds the same vector to the output of each block for every input. Bias training trains the biases a model already has, and on our models (as on most current models) there are none, or they are only inside attention. The new Appendix B.3 proves (ii) that for every change in the attention parameters, not only for the biases: a change of the attention weights cannot add the same vector to the block output for every input either. This means that bias training cannot be applied at all to 13/18 of the models we apply the shift adapter to, which have no biases, and on the other 5 (all Qwen2.5), whose biases sit only in the query, key, and value of attention, it cannot add the same vector to the output of each block as the shift adapter does.

Q2: What are the novel contributions relative to Ben Zaken et al. (2022) and Sinii et al. (2025)?

The fundamental novel contribution of our work is a new principle, the potentiality and actuality principle, that arises from the adaptation that is the same for every input. This principle shows that the capacity for a task is already in the pretrained model, and adaptation actualizes it with a change that is the same for every input, shedding new light on different subfields, including parameter-efficient adaptation, steering, and neural tickets, including the work of Ben Zaken et al. (2022) and Sinii et al. (2025).

The differences between our adaptation and Ben Zaken et al. (2022) are explained in Q1 in more detail. Compared to Sinii et al. (2025), they train one vector per layer with reinforcement learning and show that it suffices for mathematical reasoning, but they suspect that input-independence is a limitation that caps the performance. We show the opposite across a wide range of tasks. Neither work gives such a unifying picture of the principle of adaptation, nor a more general understanding of what adaptation adds, when the capacity of the base model to solve tasks via shift adapter forms, or what this implies for other fields.

In particular, our paper focuses on the potentiality principle (i.e., a capacity of the model for task-specific expertise that just needs actualization via constant shift), and investigates the following, which none of the other works do:

(i) What adaptation changes. We consistently show across 18 models from 7 architecture families and 26 tasks that the shared component alone, without any input-specific adaptation, achieves comparable performance to LoRA.

(ii) When the capacity arises. If the expertise is already in the base model, a change that is the same for every input can only actualize a capacity that pretraining has formed, so it should fail early in pretraining and succeed later. This is indeed what we find when we investigate the checkpoint of OLMo-2-7B during pre-training. Specifically, we find that the shared component of LoRA produces none of 42 writing systems at 9B tokens of pretraining, and 37 to 40 of the 42 of the full LoRA at 105B tokens (new Appendix F.11).

(iii) What structure it has. The expertise in a model is not located by one exact vector, but by any vector in a connected region. The bigger this region, the more strongly the model can produce the behavior. For example, two shift adapters trained independently, even with different supervision (the model's own answers or external translations), lie in a connected region where, if one interpolates between them, even the weakest point retains a median 99% of the gain. Furthermore, across 9 behaviors and 18 models, the size of the region correlates with how strongly the behavior is produced (within-model Spearman 0.64).

(iv) A unifying perspective. The principle connects fields that have been studied apart, as different ways of reaching the actualizing states of a model:

  • parameter-efficient fine-tuning: an update restricted to a small parameter space, or to a few vectors, suffices because it only has to actualize the capacity;
  • activation steering: the trained shift adapter is a reference for the vectors that steering extracts, and it reaches an adherence of 0.68, against 0.17 for the best vector extracted from activations;
  • in-context learning: the demonstrations reach the same behaviors through a different route, since their shared component is nearly orthogonal to LoRA's and the shift adapter’s;
  • the convergence of models: the actualizing states of different models agree, and more so between more capable models (Spearman 0.68);
  • neural thickets: an expert found by random perturbation is a perturbation that lands in the region of actualizing states, and region size follows the density of experts (Spearman 0.43).

The shift adapter is also more parameter-efficient than LoRA (0.04M against 9.2M trainable parameters on Qwen2.5-1.5B), but this is not the main point: it is the instrument that makes the principle measurable. The revised related work states this relation, and the new Appendix B.3 gives the proposition on bias training.

References

[1] E. Ben Zaken, et al. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. Annual Meeting of the Association for Computational Linguistics (ACL), 2022.

[2] V. Sinii, et al. Steering LLM Reasoning Through Bias-Only Adaptation. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.

[3] F. Yin, et al. LoFiT: Localized Fine-tuning on LLM Representations. Advances in Neural Information Processing Systems (NeurIPS), 2024.

Reviewer_2JsdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The proposed paper investigates the following question: when performing task adaptation on foundation models, how much is that adding new capabilities versus surfacing already existing ones? They answer this question by decomposing the update induced by downstream task adaptation into a per-input component and a shared component. Results show that per-input component yields little gain, while the shared component largely recovers the behaviour of LoRA task-adaptation.

The conclusion is that foundation models acquire task-specific capabilities during pre-training, and downstream task-adaptation functions merely by exposing these latent capabilities via input-independent shifts. The authors name this property the potentiality principle (or the potentiality and actuality principle in Figure 1).

Along with this principle, the paper suggests a new way of adapting a model: the shift adapter (a vector per layer) and demonstrates that it recovers the performance of LoRA or full fine-tuning.

Critical strengths

The strenghts of the paper are:

  • the good coverage over adaptation methods: LoRA, full fine-tuning and in-context learning;
  • the clear and extensive empirical setup (lots of benchmarks and models across different architectures) that answers the premise question and supports the author's conclusions.
  • unifying conceptual connections to related studies (supported by empirical evidence in the case of activation steering, as well as in the case of neural thickets).
Critical weaknesses

I want to flag a few potential problems with the paper:

1. A constant parameter shift is not an input-independent function (acknowledged in Section 3 and Lemma B.3 that a constant shift induces input-dependent effects through subsequent layers). A shift-adapter can memorize up to 200 arbitrary key-value pairs. That means you can't interpret the success of a shift-adapter as a proof of a pre-existing capability (potentiality). It cannot distinguish between actualizing latent capabilities and parameter-efficient memorization.

2. Continuing on point 1., the euclidean energy share and the cosine similarity might not be the right lens. Adaptation requires input-independent "forgetting" before absorbing the new data. This forgetting component could dominate the cosine similarity and the energy share marking the lower-norm input specific residuals. In Appendix E.5 there is a mention that as the lookup tables grow from 100 to 500 entries, the gain retained by LoRA's shared component collapses from 100% to 16%, while the shared energy share remains invariant.

3. The paper doesn't provide a chraracterization of the relationship between pretraining (or pre-adaptation) data and adaptation data that would predict apriori when adaptation on an arbitrary task reduces to an input-dependent shift. Without that it becomes circular: there was potentiality if the shift succeded. The claim "to adapt to a new task, the model just needs to be actualized" needs a condition.

Questions for authors

Given that shared energy share remains invariant even when the gain of LoRA's shared component collapses on the 500-entry lookup table (Appendix E.5) and that even random LoRA shows a 0.85 cosine similarity to its shared component (Figure 5), could the high energy share and cosin similarity reflect a high-norm, input-independent shift required to "forget" and mask the input-dependent shifts?

Do you have any hypothesis on why Mistral-7B is the only outlier where LoRA and unconstrained updates rely mostly on input-specific commponents (Section D10, but also the authors observation).

5 comments

We thank the reviewer for finding that the clear and extensive empirical setup "answers the premise question and supports the author's conclusions", for recognizing that our work offers unifying conceptual connections across areas, and for useful feedback.

Most of the reviewers' concerns rest on one point, which the reviewer states in W1: "a constant parameter shift is not an input-independent function". We agree, and we state this explicitly (in the first paragraph of Section 3). However, the misunderstanding is that what a shift adapter learns is input-independent (i.e., one vector per layer, the same for every input). Since the learned change is the same for every input, every difference between the answers to two inputs has to be computed by the frozen base model. This is fundamentally different from LoRA, full fine-tuning, and bias training. They change each layer by a matrix applied to its input, so their change consists of a part that is the same for every input and a part that differs for each input, and their success alone cannot tell whether the answer is computed by the base model or by the update. The input-independent change is what allows us to propose the potentiality and actuality principle: the capacity to answer each input correctly is already in the base model, and adaptation only has to actualize it. To make this distinction clearer, (1) the introduction now states it in the paragraph that introduces the shift adapter, and (2) Section 3 now states how this sets the shift adapter apart from LoRA, full fine-tuning, and bias training (proof in the new Appendix B.3), and that a shift adapter can also store the entries of a table, but if it is only storing entries, it would not be able to generalize to unseen samples (more details in W1).

While the shift adapter is more parameter-efficient than LoRA, this is not the main point that the paper makes. Our main contribution goes beyond yet another adaptation method, and what we propose is the potentiality and actuality principle, which reveals that the pretrained foundation model already has task-specific expertise that only needs to be actualized via a constant push, allowing us to directly study what adaptation changes, when the potentiality arises during pretraining, and what structure it has. The principle also connects fields that have been studied separately so far, such as fine-tuning, in-context learning, activation steering, representational convergence, and neural thickets, providing a unifying view on them.

W1: A shift adapter can memorize up to 200 arbitrary key-value pairs, so its success cannot be interpreted as proof of a pre-existing capability (potentiality), and it cannot distinguish between actualizing latent capabilities and parameter-efficient memorization.

We thank the reviewer for this observation, but we respectfully disagree. On the behaviors and tasks, a shift adapter succeeds because the base model already holds the capacity, not because the shift adapter stores its training items. It stores items only where the task needs new content, such as the entries of a lookup table, which we investigate in the paper. In response to the reviewer’s comment, we explain this in more detail and conduct thorough new analysis (new Appendices F.11 and E.6) that help to distinguish a capacity that the base model held before adaptation (in our terms, a capacity held in potentiality) from parameter-efficient memorization.

In particular, if the success of a shift adapter was due to the parameter-efficient memorization, we would expect the following:
(1) on a lookup table, the entries and the operation of looking up an entry would be stored together, in one vector;
(2) a shift adapter would match LoRA’s performance at every point of pretraining;
(3) a shift adapter would be correct mainly on the samples it was trained on, and would need to be trained on many samples to generalize.

We conduct new experiments to investigate these three points.

(1) We first investigate which part of a shift adapter holds the entries of a lookup table (new Appendix E.5). To separate the operation of looking up an entry from the entries themselves, we train, across 10 models, 24 shift adapters on each, each on a different lookup table, and take their shared component (i.e., the part all 24 have in common, the operation of looking up an entry). We add this shared component to the model and keep it fixed. On top of it, we train a second vector per layer, the stored content, on the entries of a new table only. The construction allows us to add the operation and the content separately. Adding the stored content alone leaves the model's answers unchanged. Adding the shared component alone makes the model invent an entry in 98% of its answers, so it looks up an entry but has no entries. Adding both components returns the correct entry of the new table in 99% of the answers. So on a lookup table, a shift adapter memorizes the entries in the stored content, while the operation of looking up an entry is not memorized: the shared component actualizes it in the base model, even for a table it was never trained on.

(2) We additionally test whether a shift adapter can reach a behavior and match LoRA’s performance at every point of pretraining (new Appendix F.11). If the success is only due to the parameter-efficient memorization, a shift adapter would work at 9B tokens as well as at 105B tokens, as LoRA does. To test this, we first use the pretraining checkpoints of OLMo-2-7B and, as the behavior, answering in 42 writing systems, that is, in languages written in their own non-Latin script, such as Thai, Kannada, or Bengali. At 9B tokens, a rank-8 LoRA already produces 40 to 42 writing systems, so the behavior can be learned at this checkpoint. However, at this checkpoint, a shift adapter produces only 7 to 8 behaviors. The shared component that LoRA learns confirms this: at 9B tokens, the shared component of the trained LoRA alone produces none of the writing systems. At 105B tokens, a shift adapter produces 38 to 41 of the 42 writing systems of the full LoRA. This shows that with the same architecture and parameter count, the behavior is there only once the base model has formed the capacity during pretraining. If the model is just memorizing, the memorization would work wherever LoRA does, at 9B tokens as well as at 105B tokens.

(3) We further test memorization directly with an experiment in which we change only the number of training examples (new Appendix E.6). On 4 models from 4 architecture families (Qwen2.5-1.5B, Llama-3.2-3B, SmolLM2-1.7B, Gemma-2-2B), we train shift adapters on 8, 32, or 80 training examples and evaluate all of them on 200 held-out examples, none of which is ever used for training. With only 8 training examples, a shift adapter already reaches 59% of its gain at 80 training examples on the traits and 68% on the transformations, and with 32 training examples 83% and 92%, all on examples it has never seen. The same experiment does detect memorization where there is content to memorize: on a lookup table, a shift adapter trained on 8 of its 26 keys answers 28% of the prompts whose key it saw and almost none (0.1%) of the others. Shift adapters trained on three disjoint sets of 8 instructions also learn the same direction: on the traits, their average does better than each single shift adapter in each of the 4 families, while on the lookup it does worse. So on the behaviors, a shift adapter learns from a handful of items a direction that holds for every input, and only on the table does it memorize the items it saw.

In summary, a shift adapter memorizes content where the task is a table of new entries, as the reviewer notes and our original paper and new experiments investigate. However, on the behaviors and tasks, it actualizes a capacity of the base model: the operation is not stored, the constant works only once pretraining has formed the capacity, and a handful of items suffices on inputs never seen in training. These experiments show that our work can distinguish a capacity that existed before adaptation from parameter-efficient memorization.

In response to the Reviewer’s comment, we (1) updated Section 3 of the revised manuscript, which now clearly states this distinction: a capacity is actualized when a constant vector suffices on inputs never seen in training, and stored when the gain holds only on the items seen in training, and (2) included new experiments that investigate this question further in Appendices E.6 and F.11.

W2: The euclidean energy share and the cosine similarity might not be the right lens, since an input-independent "forgetting" component could dominate them. In Appendix E.5, the gain retained by LoRA's shared component collapses as the lookup tables grow, while the shared energy share remains invariant.

The field share (i.e., "euclidean energy share") and the cosine similarity describe the update made during adaptation, not what the model gains from it. A component that is the same for every input, such as one that makes the model "forget" its previous behavior, could indeed dominate both without carrying the gain. We locate the gain with interventions that score it directly, not with these two measures, and these interventions establish the importance of the shared component. We further address the reviewer’s comment in two steps: (1) explain why the field share stays the same in Appendix E.5 while the gain kept by the shared component falls, and (2) why the shared component is not a forgetting component.

(1) The field share measures how much of a change is the same for every input, not what the gain depends on. We agree it cannot tell the two apart, so the paper investigates them together with two interventions that score the model's answers: keeping only the shared component, and removing it. On the lookup tables, as the table grows from 100 to 500 entries, keeping only the shared component keeps 76% and then 14% of LoRA's gain, while the field share stays about the same (0.32). The gain thus moves into the input-specific component, which holds the table entries, as expected for new content that must be stored. On tasks where no content has to be stored, keeping only the shared component keeps on average 85% of LoRA's gain across 18 models and 17 tasks. So the field share describes the change, and the interventions tell where the gain lies, which is why the paper presents them together.

(2) We interpret the "forgetting" component mentioned by the reviewer as a change that is the same for every input and suppresses the base model's previous behavior, such as its usual way of answering, so that the input-specific component can then learn the task. If this were the case, we would expect three things:
(i) keeping only the shared component would remove what was learned and keep little of the gain;
(ii) a shift adapter, which has only a shared component, could forget but not learn the task, and would stay far below LoRA;
(iii) a shift adapter would change the accuracy on the tasks it was not trained on, since forgetting is not specific to one task.

(i) Keeping only the shared component retains 85% of LoRA's gain on the tasks, rather than a small fraction. (ii) A shift adapter reaches a mean accuracy of 0.746 over 18 models and 17 tasks, against 0.745 for a rank-8 LoRA and 0.620 for the base model, so it learns the task. (iii) In a new experiment, we evaluate every shift adapter, each trained on one task, on all 17 tasks, on the 18 models (new Appendix E.7). If the shift adapter made the model forget its previous behavior, accuracy on the 16 other tasks would fall, and if it improved the model in a general way, accuracy on them would rise. On the task it was trained on, a shift adapter raises accuracy by 0.124 on average. On the 16 other tasks, accuracy changes by an average of -0.004, the same as with a random vector of the same norm at every layer. So the shift adapter changes the model only for the task it is trained on: it neither makes the model forget its previous behavior nor improves it in a hidden, general way.

Comments on the paper

1 comment

Dear reviewers

We thank you for your valuable feedback. The main concerns raised by the reviewers were the relationship to prior work and the better disentanglement between memorization and a preexisting capability of a foundation model (i.e., potentiality). We thoroughly addressed both concerns, with six new experiments, a proof, all incorporated in the revised manuscript. We also clarified what input-independence means and how our work fundamentally differs from the prior works. Below, we summarize the main points from the reviewers and our answers.

Relation to prior work. Related works restrict adaptation to a few parameters, such as biases, offsets on attention heads, a LoRA on one layer, or find experts by random search in weight space. Their updates depend on the input (new Appendix B.3 proves this for bias training), or they treat input-independence as a limitation. These works make adaptation cheaper, but none of them explains why or how adaptation works. In contrast, while our work provides a cheap way to compute the update, the main focus is that the learned change is input-independent (i.e., one vector per layer, the same for every input). This matters because it means that every difference between the answers to two inputs has to be computed by the frozen base model. This leads us to the potentiality and actuality principle, which fundamentally explains how adaptation works. It explains the results obtained by prior works in a principled way and why they succeed (new Table R1), as well as other subfields, including steering (where we show that the steering work corresponds to the shift adapter and substantially outperforms prior works), as well as neural thickets. In addition, our work shows that the size of the actualizing region correlates with the model performance and shows that the actualizing states of different models agree, and more so between more capable models. This provides a new way to look at adaptation.

Does a shift adapter memorize its training items? We show this is not the case with a range of different experiments. While our original submission already had experiments showing this, we make this important point now stronger with additional experiments. If the model was just memorizing, it would work at many points during pretraining, wherever LoRA does. In new experiments, we show that on OLMo-2-7B after 9B tokens, LoRA produces 40 to 42 of 42 writing systems, but a shift adapter only 7 to 8. After 105B tokens, the shift adapter produces 38 to 41 (new Appendix F.11). This shows that the capacity must first form during pretraining before a shift adapter can actualize it. Additional experiments also show that 8 training examples already give 59% to 68% of the gain on inputs never seen in training, showing that even with a few training examples, the model's capacity is actualized rather than memorized (new Appendix E.6). We describe these and further experiments in more detail in our responses to Reviewers 2Jsd and 6mi2 (Appendices D.2, E.5, E.6 and F.11).

Can one tell before training when the principle applies? Yes, for rule-based tasks. A shift adapter adds the same vector to every input, so the base model must already separate the inputs by what the answer depends on. A linear readout of the base model, fitted before any training, tests this. It predicts which rules a shift adapter learns (correlation 0.98), and where it fails, the shift adapter fails too (new Appendix F.12).

Does the shift adapter only make the model forget? No. It leaves accuracy on the tasks it was not trained on unchanged (new Appendix E.7).

In summary, our contribution is a novel principle of foundation models rather than yet another adaptation method. The principle explains why adaptation works, how it works, and where it arises. It connects parameter-efficient fine-tuning, steering, and neural thickets, as well as prior works reviewers mentioned, which all reach the states that actualize a capacity the model already holds.

Changes in the revision. In response to all reviewers' comments we added:

  • new Appendix B.3 with a proof on bias training
  • new Appendix E.6 on training with few examples
  • new Appendix E.7 on forgetting
  • new Appendix F.11 on when the capacity forms during pretraining
  • new Appendix F.12 with a condition for rule-based tasks
  • new Table R1 with the differences to related work
  • extended Appendix D.9 on the Mistral model
  • revised Appendices D.2 and E.5 on held-out items and lookup tables
  • revised Appendices G.2 and G.4, which keep selection apart from evaluation
  • revised abstract, introduction, Section 3, conclusion and related work

We believe that our response thoroughly addresses the concerns raised by the reviewers and kindly ask the reviewers to engage with the revision and let us know if there are any remaining concerns.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.