acceptodds
Under review as a conference paper at ICLR 2027

Signatures of semantic search in the activations of large language models

Abstract

When recalling lists of concepts (e.g., animals) during the semantic fluency task (SFT), both humans and large language models (LLMs) organise their output into clusters of related items (e.g., sea animals) that are punctuated by strategic switches between clusters. In humans, this pattern can be explained by a semantic foraging process, whereby distinct neural and behavioural signatures accompany within-cluster production (“exploit") and between-cluster switching (“explore"). Whether LLMs likewise represent these two search regimes within their internal states is unknown. Here, we apply a range of mechanistic interpretability techniques to provide evidence for semantic foraging in LLMs. In Study 1, we show that concept-level activations predict switching in SFT. First, we find that switching coincides with elevated next-token uncertainty (low token probabilities). We then show that these switch-associated differences in activations emerge in the early-middle layers by using the recent Jacobian lens (J-lens) approach, which maps intermediate-layer residual-stream representations to token-level activations. Moreover, the probability of switching rises as the set of strongest J-lens activations (the J-space) becomes depleted of items from the category currently being produced, analogous to explore-exploit decision-making during human semantic foraging or animal patch foraging. We then show that middle-layer J-lens activations of abstract category-related labels (e.g., “water") increase in anticipation of switching into that category. We confirm that these intermediate representations causally influence switching by using them to derive steering vectors that target switching into particular categories. In Study 2, we identify generic residual stream directions that are activated during and in anticipation of switching. By steering model activations along anticipatory switching directions, we bias increased or decreased rates of switching. Our paper extends the semantic foraging framework to artificial intelligences and provides evidence that LLMs maintain distinct representational signatures for exploration and exploitation as they verbalise conceptual information.

Primary area:
interpretability and explainable AI
Keywords:
cognitive science, search, large language models, semantics, semantic search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2cxGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether LLMs exhibit exploration–exploitation dynamics during semantic fluency tasks. The authors use J-Lens to analyze concept activations and activation steering to manipulate category-switching behavior.

Critical strengths

The motivation is interesting, particularly the attempt to connect human semantic foraging with internal representations in LLMs. However, the contribution remains largely observational and relies on combining existing techniques. The paper lacks sufficient mechanistic specificity, methodological novelty, and rigorous causal validation. Overall, I do not believe the work meets the acceptance standard of ICLR.

Critical weaknesses

I find the methodological novelty very limited. Semantic foraging is an established framework, while J-Lens and activation steering are existing techniques. The paper mainly combines these approaches in an animal-listing task without introducing a new computational mechanism or interpretability methodology.

The proposed mechanistic interpretation is insufficiently supported. Correlations between token uncertainty, category-related activations, and switching do not establish an independent exploration–exploitation mechanism. Even successful steering may simply alter the generation probabilities of category-related tokens rather than manipulate an underlying exploration state.

Questions for authors

Where are your contributions in Section 1?

1 comment

We thank Reviewer 2cxG for their comments, which have prompted several new analyses and a clearer framing of our contribution. First, our paper is intended as an empirical contribution, not a methodological one. As such, it was not a goal of our paper to develop new mechanistic interpretability techniques. Rather, our work brings recently developed interpretability tools to bear on an important question in cognitive science and AI: how do LLMs represent the retrieval of semantic information during a canonical semantic search task? Semantic fluency is an important flashpoint for studying memory retrieval processes in humans, and, to our knowledge, our paper is the first to apply mechanistic interpretability techniques to study the same dynamics in LLMs performing the SFT. Our results provide evidence for a number of representational dynamics in LLMs that support both foraging and associative accounts of retrieval, accounts originally developed to explain human semantic memory. We now outline our contributions more explicitly in the introduction. We agree that associations alone do not establish a search mechanism. This is why we paired each of our correlational analyses with a causal manipulation; each of these were effective (Fig. 4). While negative steering in Study 2 was weaker, and we now analyse why in detail, as described in our other replies. The only analysis that was not paired with a causal manipulation in our submitted paper was the in-category J-space supply. We have now added this manipulation. Depleting the J-space of the current category’s unnamed items increases switching, and replenishing it decreases switching. Finally, we also find that token entropy predicts switching and have included this result in our paper. All of these results provide useful points of comparison with human semantic search.

We agree that our results do not establish independent exploration and exploitation states. Rather, our paper provides evidence for a single, category-general direction whose loading builds up in advance of a switch, which may be analogous to a pattern of neural activation seen to increase when a person is building up to a switch (Lundin et al., 2023). We have revised our language throughout to reflect this. Regarding token probabilities, it is expected that the generic direction identified in Study 2 alters token probabilities, as any intervention on the residual stream ultimately does. Altering token probabilities would not discount a foraging account, as any internal signal that drives switching must eventually do so. The relevant question is whether the direction acts only by directly promoting particular category-related tokens (rather than increasing the likelihood of switching generally). Two results argue against this. First, as shown in Fig. 6, the direction predicts switching between categories that were held out when the direction was fitted, so it is not tied to the tokens of particular categories. Second, to test the reviewer’s concern directly, we removed from the direction its entire component in the span of the J-lens directions of every animal name and category label in our norms. This span accounts for a substantial part of the direction, more than for a random direction. Nonetheless, steering with the remaining token-orthogonal component still increased switching relative to a random direction at every tested layer in Gemma-9B and at most tested layers in Llama-8B, without degrading the model’s output. The switching direction, therefore, does not act simply by promoting specific category-related tokens. These analyses are now reported in the Appendix.

We hope that these additions, together with the new analyses described in our responses to the other reviewers (associative controls, token-entropy and logit-lens baselines, and a replication in the food domain), address the reviewer’s concerns about the soundness and specificity of our conclusions.

Reviewer_dsEfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper uses a well-established paradigm from cognitive science — semantic fluency — or simply listing members concepts of categories to test for the organization and deployment of semantic knowledge in a set of open weight language models. Specifically, the authors search for evidence of foraging behaviors observed in human minds — that of exploiting subcategories before exploring other neighbouring subcategories. The authors bring to bear several interpretability techniques including tracking token probabilities, J-lens activations, and steering methods to predict the likelihood that models will ‘switch’ to different subcategories (e.g. pets vs. sea animals from Troyer et al.).

Overall, I found the paper enjoyable to read and believe both the core ICLR audience and cognitive scientists will find something interesting to take away from this paper. There are some conceptual issues worth clarifying, which I will lay out below.

Critical strengths
  • The task paradigm is well established, conducted clearly with open models and is building on taxonomies from Troyer et al, which have also been well validated to the best of my knowledge.
  • The paper is in conversation with the latest techniques in the field and is using the host of interpretability tools available to resolve the core question of the paper.
  • The authors are thorough in their reporting of statistics, and the figures although a bit small and in need of some revision are generally helpful and illuminating.
  • The topic stands to be of interest to core AI researchers and cognitive scientists.
Critical weaknesses
  • I think the core conceptual issue is that the authors are seeking to validate one established hypothesis (that models perform foraging) while not considering strong alternative accounts. This is akin to not having strong baseline comparisons. Given that autoregressive models by their very nature are learning next-token prediction, could a simpler associative account not explain much of the behavior we see? Such accounts would also not necessarily be incongruous with many of the interp results the authors present. I think in terms of the framing, this is a core issue the authors should at the very least address. Since the Troyer items are also very likely in the training corpora, how can we fully discount the associative/memorization hypothesis here?
  • While it is interesting to see the J-lens analysis, I would have liked to see much simpler analyses based on internal activations that also comport with the core findings. Basically the issue is — to what extent are these results contingent on the use of J-lenses? For example, could token entropy not be just as effective?
  • Relatedly, as interesting as the depletion result is, the appendix makes it seem that it is a much weaker result than I initially took away from the main text. The category specific effect does not arise for Qwen at all and only for a few blocks in Gemma. What are to make of this finding?
  • I feel similarly about the steering results, which admittedly the authors also note to be mixed. The fact that negative steering increasing swithcing for the 9B gemma and llama at times seems strange. Rather than less effective, it seems that steering did not work. I almost recommend shelving much of the steering analyses though it does help contribute to a causal story. I think there is plenty to work with given the other analyses.
  • This is the least critical weakness, but this paper does only look at one domain (animals) and does not vary the prompt space. This does undercut the generality of the findings to a degree.
Questions for authors

Here are some clarification questions and suggestions for things to improve the paper


  • How would the authors distinguish their foraging interpretation from simpler random-walk/associative accounts?
  • Could the authors include simpler J-lens alternatives, such as simply looking at token entropies?
  • It is worth showing the unsteered probability of switching into each target category, and how steering affects the rate of valid outputs.
2 comments

We thank Reviewer dsEf for their positive evaluation of our manuscript and their constructive feedback. In response to their review, we have implemented a number of changes that we feel have greatly strengthened our contribution. We agree with the reviewer that our initial paper was focused on semantic foraging, without including much discussion on associative processes. First, we note that the semantic foraging framework generally does include associative processes as being important during search: semantic relations between concepts define the structure and accessibility of one’s internal search space and associative retrieval can support local exploitation (Hills, Todd, and Jones, 2015). We have expanded the discussion on this in our introduction. The question relevant to our paper is, then, whether switching between patches involves additional mechanisms, beyond associative processes alone (as in a random-walk model). As such, we have now performed analyses to test the role of associative processes in switching. Using a pretrained word embedding (fastText), we computed several measures relevant to associative processes: the similarity of each list item to the previous item, and its similarity to the unnamed items of its own category and of other categories. We then entered these measures as predictors of switching into logistic regression models, alongside J-space depletion, the number of unnamed category members, cluster length and list position. Associative similarity does predict switching, most clearly an item’s similarity to the remaining members of its own category is negatively associated with switching. However, it does not absorb the effect of J-space depletion, which remained a significant predictor of switching in all models and layer block combinations bar one, with a larger coefficient than any of the associative measures. These findings are compatible with the idea that foraging and associative processes both play a role in switching in LLMs (Hills, Todd, and Jones, 2015). Indeed, our finding that category-level concepts receive higher activations during, and in advance of, switching into them is also in line with associative processes. Our range of findings reporting changes in model activations before and during switching offer evidence against the countervailing random-walk account. Overall, our manuscript provides evidence for both, non-mutually exclusive, accounts and has been reframed to state this more clearly. Regarding the Troyer norms appearing in the training data, we used an extended version of the Troyer norms that includes some additional subcategories and is currently unpublished. Moreover, we have now repeated Study 1, and Study 2, with Gemma-9B in the food domain, using a separate set of norms (Zemla et al., 2020). While these categories may also appear in training data, this replication adds convergent evidence for our conclusions (see our further response to the final point below). We thank the reviewer for raising the point about our initial manuscript's dependence on the J-lens. We have now examined token entropy as a predictor of switching, and compared our J-lens results with those obtained with the older and simpler logit lens. Elevated next-token entropy also predicts switching, and we have included this result in Figure 2. We also obtain similar results with the logit lens as with the J-lens (as conceded in the original J-lens publication), providing evidence that our results do not depend on applying the Jacobian transformation to model activations. Finally, we note that Study 2 does not rely on any lens: the generic switching direction is identified from raw residual-stream activations, and it both predicts switching and, when used for steering model activations, influences it.

The appendix table tested whether J-space depletion is more strongly associated with switching than overall category depletion. However, a more relevant question is whether J-space depletion predicts switching over and above category depletion. We have therefore fitted models that include both measures, alongside cluster length, list position, and the associative measures described above. When category depletion is held constant, J-space depletion significantly predicts switching for all tested models and layer blocks, bar a single block in Llama-8B, and its coefficient is, on average, larger than that of category depletion. We have also conducted new causal manipulations in which we empirically deplete or replenish the J-space with the unnamed members of the current category (so far in Gemma-9B). Consistent with the idea that J-space supply is causally important for switching, depleting the J-space increased the probability of switching at nearly every layer, while replenishing it decreased the probability of switching from the middle layers onwards. We agree that our negative steering results are less convincing. However, we note that positive steering was generally highly effective across models and layers, consistent with the identified direction contributing to switching behaviour. We have therefore retained them, but investigated why negative steering was weaker and why it could sometimes increase switching. These increases reflect several effects of negative steering that were not originally noted in our study. In Llama, strong negative steering in earlier layers could lead the model to chain animal names by their spelling instead of their meaning, for example listing names that begin with the same letter. Because such names usually come from different categories, these transitions were scored as switches. But from an expanded associative-cluster perspective, "animals beginning with 'b'" could be a legitimate cluster, too (so this would correspond to the expected decrease in switching). We have included further analyses on these unexpected behaviours in the appendix. Moreover, perturbing the residual stream in any direction tends to increase switching, and relative to this control negative steering generally reduces switching. Finally, in Gemma and Llama, negative steering was most effective when the current category was close to depletion, where the model is most likely to switch, whereas positive steering increased switching regardless of how much of the category remained. We have revised the text to explain these factors. Finally, we have now replicated our findings in the food domain with Gemma-9B. Most of our core findings replicate and are now shown in the appendix. Some effects are weaker for food, and we report these in the revised manuscript. We have not yet varied the prompt, and agree that this would further test the generality of our findings; we intend to repeat the study with paraphrased prompts and also report these results.

Reviewer_qJJKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how LLMs represent semantic exploration and exploitation during the Semantic Fluency Task (SFT). Study 1 uses J-lens analyses to link switching to fewer unmentioned items from the current category in J-space and rising activation of labels related to the next category. Study 2 analyzes residual-stream activations to identify general switching-related directions and shows that steering along them can increase or decrease switching rates, with effects varying across models and layers.

Critical strengths
  1. The paper connects human semantic foraging with LLM internal representations through an interesting research question. Combining the activation analysis with steering experiments offers insight into how these representations contribute to semantic switching.
  2. The paper develops clearly from item- and category-level analysis in Study 1 to general switching-related directions in Study 2, supported by informative figures.
Critical weaknesses
  1. The abstract claims that LLMs maintain distinct representational signatures for exploration and exploitation, but the connection to the findings in Sections 3 and 4 is not entirely clear. In particular, does the generic switching direction in Section 4 represent both behaviours through higher and lower activation levels?
  2. The experiments cover several LLMs but only test animal-name generation. It remains unclear whether the representations and steering effects exended to other semantic domains.
  • Minor: some figures are discussed out of order in the text (e.g., Figure 1B and Figure 2).
Questions for authors
  1. What drives spontaneous semantic switching in LLMs? Is there any evidence for a distinct semantic foraging mechanism?
  2. Figure 2D shows that models switch more often when fewer unmentioned items from the current category remain in J-space. Does this reduced supply encourage switching, or simply reflect that the model is already shifting toward another category?
  3. In Figure 5E and 5F, positive steering generally increases switching, but negative steering is less consistent. What might explain this difference?
1 comment

We are pleased that reviewer qJJK22 sees value in using mechanistic interpretability to study semantic foraging in LLMs. We agree that our study provides limited support for the broader claim that exploration and exploitation have distinct representational signatures. Our findings support a more specific claim about switching behaviour. As the reviewer suggests, model activations load more strongly onto the generic switching direction identified in Study 2 as a switch approaches. We have revised the manuscript throughout to reflect this distinction. This interpretation is consistent with studies of human semantic foraging, which find that changes in activation in particular brain regions predict switching. Such findings can be explained without positing qualitatively different representational dynamics during clustering and switching.

In Study 1, we originally showed that J-space supply was associated with switching probability. At the reviewer’s suggestion, we have added a complementary causal manipulation in which we deplete or refill the J-space at different model layers (Fig. 5B). During a cluster, we decrease or increase the J-lens activations of unnamed items in the current category and measure whether the next item stays in that category. Across the models tested, depleting the J-space increases the probability of switching to a new category, while refilling it reduces that probability. These results are consistent with our earlier token-level steering findings. By targeting the supply of unnamed items in the current category, the new manipulation provides more direct evidence that J-space supply influences switching behaviour.

We also agree with reviewer qJJK22 that the difference in effectiveness between positive and negative generic steering warranted further investigation. Our additional analyses identify several factors that help explain why the effects of negative steering were less consistent.

First, the effectiveness of negative steering depends on how many readily nameable items remain in the current category. For the large Gemma and Llama models, negative steering tended to delay switching most effectively when few such items remained and the probability of switching was already elevated. When many items remained, the model was already unlikely to switch, and negative steering had little further effect. Positive steering increased switching regardless of how many items remained. This pattern is consistent with the generic direction reflecting an approaching switch. Subtracting that direction can delay a switch as it becomes likely, but has little effect while the model is likely to continue naming items in the current category. Qwen showed a different pattern. Negative steering reduced switching regardless of how many items remained. In the food domain, where the switch rate was approximately twice that of the animal domain, negative steering was also generally effective in the largest models tested.

Second, negative steering in Llama-3.1-8B, the model with the least consistent response to negative steering, sometimes led it to list animals whose names began with the same letter, as in crocodile, cat, crab. Negative steering can therefore encourage clustering by spelling. Consecutive items linked by their initial letter often belong to different Troyer categories, so these transitions are scored as semantic switches.

Finally, our noise control showed that perturbing the residual stream along a random direction of matched magnitude also tended to increase switching. Relative to this control, negative steering generally reduced switching across models. These analyses are now reported in the appendix.

We also tested whether the findings of our original animal-domain study generalised to the food domain, including in Gemma-9B. Although food categories are less clear-cut, most of our core findings replicated. Items produced at a switch showed lower J-lens activation before emission than items produced within a cluster. Category labels were activated both while the model named items from the corresponding category and in anticipation of a switch into it. Switching became more likely as the supply of unnamed in-category items in the J-space declined. Steering along the averaged J-lens directions of items in a target category also made the next item more likely to belong to that category. This replication also makes it less likely that exposure to the animal-category norms during training fully explains our findings.

Given the compute time available during the review period, we generally limited these additional analyses to the three largest models. We were unable to run the J-space depletion and refilling manipulation in the food domain during this period. Where applicable, we intend to extend these analyses to the remaining models and domains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.