acceptodds
Under review as a conference paper at ICLR 2027

When Does a Novel Form Become a Word? Causal Evidence for Reusable Lexical Function in Frozen Language Models

Abstract

When does a novel form begin to function like a word? Inferring its meaning from context is not enough: lexical function emerges when the form itself becomes a cue for reusing semantic information inferred from its prior usages. We study this transition from contextual evidence to lexical function in frozen language models. Across Qwen3-8B and Gemma-3-4B, reversing the meaning assignments of two novel forms reverses which form the model selects for the same held-out situation. We then causally intervene on internal states produced by the preceding usages, separating their influence from the original examples. Transplanted states carry semantic influence from the source context into a new judgment, with a preferential effect on the form with which that meaning was associated. The same states also guide judgments of previously unseen usages, even when the source definition or concept label is no longer presented. This lexical function depends on how the relation was formed, not merely on which semantic material appeared in context. Holding source sentences, order, length, and form frequency fixed, more consistent form–meaning assignments produce stronger transferable semantic influence; subsequent target evidence then reduces the advantage established by stronger formation. Together, these results show that implicit usages do more than make a novel form interpretable: they organize usage-derived semantic information into a relation that can be reused through the form itself. We call this history-dependent, reusable, and evidence-sensitive form–meaning relation a revisable lexical hypothesis.

Primary area:
interpretability and explainable AI
Keywords:
in-context learning, lexical induction, causal representation analysis, mechanistic interpretability, language model memory, novel word acquisition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MPw8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how frozen language models establish form–meaning associations for novel words through contextual usages and whether such associations can be reused. The authors employ reciprocal meaning assignments, activation transplantation, and held-out semantic judgments to examine whether semantic information transfers to new tasks without the original usage contexts. Experiments on Qwen and Gemma further investigate the effects of form–meaning consistency and subsequent evidence. The authors characterize the resulting history-dependent, transferable, and evidence-sensitive relation as a revisable lexical hypothesis.

Critical strengths

The experimental design is reasonably rigorous. Rather than relying solely on behavioral evidence of novel-word learning, the authors use interventions on pre-query activations to investigate whether semantic information is available before the query. In particular, the changed-form versus unchanged-form comparison and the donor versus random source control help distinguish general activation transfer from form-related semantic influence.

Critical weaknesses

My main concern is the remaining gap between the definition of lexical function and the experimental evidence. The authors transplant residual states from previous usages and show that these states can influence subsequent semantic judgments. However, the transplanted activations contain rich contextual information rather than an isolated form–meaning binding representation. Therefore, the results demonstrate that semantic information derived from previous contexts can be reused, but they do not clearly establish a lexical representation distinct from general contextual information integration. In particular, the experiments do not directly disentangle the transfer of semantic content from the transfer of the form–meaning relation itself. Both mechanisms could potentially produce similar output changes. I acknowledge that the authors define a lexical hypothesis functionally rather than as a localized dictionary entry. Nevertheless, even under this operational definition, stronger evidence is needed to establish selective access through the associated form.

Questions for authors

see above

Reviewer_5NfcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when a novel form acquires reusable lexical function in frozen language models. Using novel forms associated with concepts from CHIME, the authors first demonstrate that Qwen3-8B and Gemma-3-4B can infer form–meaning associations from contextual usage and adapt their predictions when these associations are reversed.

Critical strengths
  • Causal investigation of reusable semantic information. : This paper designs an evaluation of novel form learning by transplanting internal states into a target computation without the source examples. This provides useful causal evidence that information derived from previous usages can influence subsequent semantic judgments.

  • Multiple complementary experiments and controls.: This paper investigates semantic transfer, form selectivity, novel-usage generalisation, binding consistency, and the influence of new evidence. In particular, the controlled manipulation of form–meaning consistency while preserving source sentences and form frequency is a useful experimental design.

Critical weaknesses
  • The difference between lexical function and general in-context learning is not established.: The results show that internal states carry semantic information and that this information influences subsequent judgments. However, it remains unclear which findings specifically establish lexical function beyond general in-context learning. For example, stronger transfer after more consistent form–meaning assignments may also be explained by reduced uncertainty in contextual meaning inference. The paper would benefit from a clearer distinction between the lexical function and these alternative explanations.

  • The methodology and evaluation measures are difficult to follow.: Several important details are distributed across the main body and appendices, making the experimental design difficult to reconstruct. In particular:

    • The definition and construction of the random-control sources are not sufficiently explained in the main body. Appendix B.4 specifies fixed concept-index settings, but the rationale for these settings and their effect on semantic similarity are unclear.
    • The construction of target-evidence examples, particularly at (d=8), requires clarification given the four demonstrations and four held-out usages described per concept.
    • Table 3 reports attenuation and target-evidence slopes, but their computation is not clearly specified in Section 4.3.
Questions for authors
  • Lexical function vs. in-context learning. Which experimental findings distinguish the proposed lexical function from general in-context learning?

  • Last-occurrence intervention. Does the history effect observed at the last occurrence generalise to earlier occurrence positions?

Reviewer_ogAnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether language models form reusable lexical representations when learning novel words from their usages. The authors argue that models can form lexical hypotheses that are associated with particular forms and revised by subsequent evidence. They test this by transferring hidden states from earlier usages into new contexts and measuring their effects on semantic judgments. The results show that information inferred from earlier usages can be reused in new contexts, with its influence depending on the consistency of previous assignments and subsequent evidence.

Critical strengths

I find it interesting that language models can associate a novel word with semantic information inferred from its usages, and reuse this information in new contexts. In particular, the authors show that the transferred information preferentially affects the corresponding word, rather than simply changing the model's general semantic judgments. The experiments on learning history further show that earlier usages affect the information carried by later occurrences of the same word.

Critical weaknesses

My main concern is that the novelty and significance of the findings could be better clarified in relation to existing work on contextual learning.

Some works i have seen have studied related topics, such as language models binding entities to attributes in context [1] and transfering information inferred from demonstrations to new queries [2]. This paper extends these findings by showing that semantic information associated with a novel word can be reused in subsequent judgments. While I find the form-selective reuse interesting, I still find it is not completely clear what additional understanding of lexical learning this provides beyond existing contextual learning mechanisms.

In particular, the authors aim to distinguish the formation of reusable lexical hypotheses from interpreting earlier usages at query time (Section 1). However, the experiments do not clearly distinguish these possibilities. The model could simply retain information about earlier usages and combine it when answering the question. The findings on learning history and subsequent evidence also seem consistent with this explanation.

I would appreciate it if the authors could clarify how their findings differ from these existing accounts and what additional insights they provide into lexical learning.

[1] Feng and Steinhardt (2024). How Do Language Models Bind Entities in Context? ICLR

[2] Hendel et al. (2023). In-Context Learning Creates Task Vectors. Findings of EMNLP.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.