- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether LLMs exhibit exploration–exploitation dynamics during semantic fluency tasks. The authors use J-Lens to analyze concept activations and activation steering to manipulate category-switching behavior.
Critical strengths
The motivation is interesting, particularly the attempt to connect human semantic foraging with internal representations in LLMs. However, the contribution remains largely observational and relies on combining existing techniques. The paper lacks sufficient mechanistic specificity, methodological novelty, and rigorous causal validation. Overall, I do not believe the work meets the acceptance standard of ICLR.
Critical weaknesses
I find the methodological novelty very limited. Semantic foraging is an established framework, while J-Lens and activation steering are existing techniques. The paper mainly combines these approaches in an animal-listing task without introducing a new computational mechanism or interpretability methodology.
The proposed mechanistic interpretation is insufficiently supported. Correlations between token uncertainty, category-related activations, and switching do not establish an independent exploration–exploitation mechanism. Even successful steering may simply alter the generation probabilities of category-related tokens rather than manipulate an underlying exploration state.
Questions for authors
Where are your contributions in Section 1?
We thank Reviewer 2cxG for their comments, which have prompted several new analyses and a clearer framing of our contribution. First, our paper is intended as an empirical contribution, not a methodological one. As such, it was not a goal of our paper to develop new mechanistic interpretability techniques. Rather, our work brings recently developed interpretability tools to bear on an important question in cognitive science and AI: how do LLMs represent the retrieval of semantic information during a canonical semantic search task? Semantic fluency is an important flashpoint for studying memory retrieval processes in humans, and, to our knowledge, our paper is the first to apply mechanistic interpretability techniques to study the same dynamics in LLMs performing the SFT. Our results provide evidence for a number of representational dynamics in LLMs that support both foraging and associative accounts of retrieval, accounts originally developed to explain human semantic memory. We now outline our contributions more explicitly in the introduction. We agree that associations alone do not establish a search mechanism. This is why we paired each of our correlational analyses with a causal manipulation; each of these were effective (Fig. 4). While negative steering in Study 2 was weaker, and we now analyse why in detail, as described in our other replies. The only analysis that was not paired with a causal manipulation in our submitted paper was the in-category J-space supply. We have now added this manipulation. Depleting the J-space of the current category’s unnamed items increases switching, and replenishing it decreases switching. Finally, we also find that token entropy predicts switching and have included this result in our paper. All of these results provide useful points of comparison with human semantic search.
We agree that our results do not establish independent exploration and exploitation states. Rather, our paper provides evidence for a single, category-general direction whose loading builds up in advance of a switch, which may be analogous to a pattern of neural activation seen to increase when a person is building up to a switch (Lundin et al., 2023). We have revised our language throughout to reflect this. Regarding token probabilities, it is expected that the generic direction identified in Study 2 alters token probabilities, as any intervention on the residual stream ultimately does. Altering token probabilities would not discount a foraging account, as any internal signal that drives switching must eventually do so. The relevant question is whether the direction acts only by directly promoting particular category-related tokens (rather than increasing the likelihood of switching generally). Two results argue against this. First, as shown in Fig. 6, the direction predicts switching between categories that were held out when the direction was fitted, so it is not tied to the tokens of particular categories. Second, to test the reviewer’s concern directly, we removed from the direction its entire component in the span of the J-lens directions of every animal name and category label in our norms. This span accounts for a substantial part of the direction, more than for a random direction. Nonetheless, steering with the remaining token-orthogonal component still increased switching relative to a random direction at every tested layer in Gemma-9B and at most tested layers in Llama-8B, without degrading the model’s output. The switching direction, therefore, does not act simply by promoting specific category-related tokens. These analyses are now reported in the Appendix.
We hope that these additions, together with the new analyses described in our responses to the other reviewers (associative controls, token-entropy and logit-lens baselines, and a replication in the food domain), address the reviewer’s concerns about the soundness and specificity of our conclusions.