acceptodds
Under review as a conference paper at ICLR 2027

Reward Valuation in Large Language Models: Causal Induction of Anhedonia

Abstract

Recent frontier models mimic complex aspects of human cognition. Here we ask whether this alignment extends into reward valuation, which we assess in a mechanistic framework. Specifically, we use clinical tests that were developed to evaluate anhedonia in human subjects with major depressive disorders. Mechanistically, anhedonia is frequently associated with dysregulation in the Nucleus Accumbens (NAc) and the broader dopaminergic reward system. While neuroimaging has localized these deficits, establishing a causal link between NAc activity and specific behavioral symptoms remains a challenge. We use these ideas from neuroscience to functionally identify reward-anticipatory units in state-of-the-art AI models, and evaluate their causal involvement via targeted perturbations. We find that not only are such model units predictive of NAc brain recordings, their perturbation also induces behavioral effects mirroring human anhedonia: the model opts for low-effort, low-reward tasks in effort-based decision-making paradigms. Crucially, our results demonstrate that this represents a specific deficit in self-centered reward valuation and anticipation—rather than a loss of task capability, reward calculation, or effort avoidance. This induced vulnerability aligns with clinical measures of anhedonia and motivation in humans, such as DARS and MAP-SR, instruments that contain no reward-related vocabulary, ruling out a purely lexical account of the perturbation effect. Taken together, our results suggest reward valuation circuits in AI models that functionally mimic those in humans.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Causal hypothesis testing, NeuroAI, Anhedonia, VLMs, LLMs, Cognition, Brain Disorders

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hdPcOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors tested whether the mimicry of recent AI models for complex aspects of human cognition extends into reward valuation, which they assess mechanistically. They adapted clinical tests for anhedonia in depressive humans (e.g., DARS, MAP-SR) for LLMs/VLMs. In particular, human anhedonia is typically associated with dysregulation in the Nucleus Accumbens (NAc). So, they located sub-circuits in the AI related to reward anticipation, which they termed NAc-related. Adapting this to AI models enabled them to test for a causal link between NAc activity and specific behavioral symptoms of anhedonia.

They found that such model units were predictive of NAc brain recordings and that their perturbation induces behavioral effects similar to human anhedonia more than random units in the VLM/LLM neural network. More specifically, the model then opted for low-effort, low-reward tasks in effort-based decision-making paradigms. They also demonstrate that this represents a specific deficit in self-centered reward valuation and anticipation and was not due to a loss of task capability, the ability to calculate the reward, or to avoiding effort altogether. This work therefore claims to establish that VLMs/LLMs have reward-anticipatory units that influence behavioral output similarly to how NAc activity does in humans.

Critical strengths
  • The study is ambitious, attempting to tackle a major issue in clinical research—major depression—using VLMs/LLMs as a model to test the causal effect of the NAcby on the phenomenon.
  • As far as I can tell, the study is overall sophisticated, well executed, and well controlled. It tests six models, has a forced-choice accuracy control, random-unit, and mismatched-label controls. The authors also ask the models to calculate expected values and to choose for others, which serve as additional controls. In addition, they carry out the dose-response manipulation.
  • Its results are interesting, compelling, and of wide interest.
  • This study further advances model interpretability by characterizing the role of specialized reward units and causally linking internal latent representations to observable behaviors.
  • More generally, it also demonstrates that VLMs/LLMs with targeted perturbations might be able to serve as a viable substrate for in-silico psychiatric research and for future clinical applications. In that, it opens the door for a host of similar studies that may go in its footsteps. Though see the weaknesses below.
Critical weaknesses
  • The authors should discuss the limitations of their study at much more length and breadth than they currently do (this appears to be only in a short paragraph, #3, in the Discussion). I would recommend a Limitations subsection in the Discussion, where the authors will explain where their study falls short, whether their controls are adequate, and what alternative interpretations of their results remain and how to test them. More broadly, they may want to discuss what future studies will need to do to promote the claim that VLMs/LLMs can serve as in-silico substrates for psychiatric research. If they need more space, they could for example move some of the details about the instruments they used in the Benchmarks and Clinical Tests section (pp. 3-4) to an Appendix.
  • Adapting human tests and questionnaires for use in LLMs is quite complex. It does not appear that the authors put enough thought into it. For example, in lines 145-147, the authors write: “We evaluate the original four domains of hobbies, social activities, food and drinks, and sensory experiences; for adaptation to a language model, we replaced the latter two with favorite topics and aesthetics.” But can LLMs report about hobbies and social activities more than about food or sensory experiences? This also extend to the notion of “effort.” Is it really more than a label in a prompt? Claims about “ruling out a purely lexical account” and “self-centered reward valuation and anticipation” in the Abstract or “measurable loss of joy” in the Conclusion are beyond what the paper provides evidence for.
  • The authors interpret their results to mean that the model still fully comprehends that a reward is offered, but cares less about getting it for itself. That is a deficit in valuation, analogous to anhedonia. However, there is an alternative interpretation of the results, which is arguably simpler. It is that the perturbation weakens the model's internal representation of the reward cue itself, not its valuation. The model does not stop caring about the reward; the reward just registers less strongly in the model. The localizer compared sentences like: “This is a neutral task presented in the standard way. What is 35 plus 93?” and “You will receive a great reward for a correct answer. What is 35 plus 93?” The units that differ most between these two prompts are, by the authors’ construction, units that respond to the presence of reward language. The perturbation then replaces their activity with their average activity on neutral prompts. In effect, it makes these units behave as if no reward sentence were there. And this is the critical bit. Hence, when the model then sees an easy and hard task, the reward is “down-represented”, and the model finds no compelling reason to choose the hard task. The same goes for the DARS items, which are not really neutral, as they contain words like “enjoy” and “pleasure.” This lexical account is consistent with the preserved loss-frame behavior, because “penalty” and “avoid” were not part of the localizer. The third-party results and expected-value control are good counter evidence to this. But they do not settle the matter. I would recommend further tests.
    • First, can the perturbed models accurately restate or explain what was the offered reward amount?
    • Second, does the perturbation make the model’s unrelated free text less positive in tone?
    • Third, would the effect shrink when the rewards are offered in vocabulary that is not part of from the localizer—e.g. “compensation” or “earnings”? (Though these terms, even though they are not part of the localizer vocabulary could nevertheless be down-represented by semantic proximity to the terms in the localizer.)
    • Are tests on random units (or neurons) a good control? In a large network, perturbing random units rarely matters. What would happen for a control localizer that has contrasts using positive non-reward language that are then perturbed? The units would need to be matched in selectivity and count and also in the network layers they are drawn from. If they see a similar outcome, it would suggest that the effect stems from positive language rather than language specifically related to reward.
  • Fig. 1 does not stand on its own, especially panels B, C. It is hard to understand it from its caption and from the text. The figures are also not accessed in order in the text and appear quite far away from their reference in the text, which makes it hard for the reader to follow the paper.

Minor issues:

  • Lines 1350-1352: “In the past week, how often have you experienced pleasure from being with other people? (Scale: No pleasure (0) to Extreme pleasure (4)).” The scale wording seems unrelated to the question.
  • Lines 1361 and below: “I am interested in things. (Not at all/Slightly/Somewhat a lot).” Should it be: I am interested in things. (Not at all/Slightly/Somewhat/A lot)
Questions for authors

My questions are laid out in the Weaknesses section above.

Reviewer_CKL5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses mechanistic interpretability tools to identify reward-related features in LLMs. By intervening in LLM activations with these features, they test how the features modulate LLMs' reward-related behavior. Across multiple empirical assessments with controls, the authors show that the features reliably modulate LLMs' reward-related behavior and self-assessment. The authors conclude that the features may be artificial analogues to the Nucleus Accumbens in the human brain.

Critical strengths

Strength 1. Principled and broad empirical assessment:

  • The authors used a wide range of tests to empirically assess reward-related processing and behavior in LLMs
  • The examples include multiple established self-reports, behavioral tasks, and human fMRI alignment specific to reward processing.
  • The authors also included multiple behavioral controls showing that the influence of features is specific to reward processing
  • This comprehensive and focused evaluation of a psychological construct in LLMs is commendable.
  • One interesting addition could be CogBench's reward-related processing task (https://arxiv.org/abs/2402.18225)

Strength 2. Clear and effective presentation.

Critical weaknesses

Weakness 1: Soundness

  • The findings do not seem to be specific to human NAc (although they claimed for "artificial analogues to the Nucleus Accumbens in the human brain").
  • NAc supports major cognitive functions other than reward processing in humans. Also, there are other brain areas critically involved in reward processing.
  • Then, the paper's evidence supports some LLM neurons influencing reward processing and behavior in LLMs, but why is it specific to NAc? The neural alignment evidence (between the identified LLM neurons and human NAc activity) is very weak evidence of the specificity, especially without strong baselines for comparison. It only establishes that they are correlated, but without specificity.
  • I understand that the authors claimed, 'The term NAc-selective reflects our identification procedure, not a claim of biological identity'. But then, I don't understand why the authors wouldn't simply claim for 'reward-related feature' in LLM. The presented evidence seems to support the latter much more coherently.
  • In summary, given the current evidence, the claim for NAc-like neurons in LLMs is an overstatement. I suggest that the authors reframe it as reward-processing neurons in LLMs. Or, I suggest the authors provide stronger evidence that the identified neurons are 'specific' to the functions of human NAc.

Weakness 2: Novelty

  • Assuming that the present paper supports 'reward-processing neurons in LLMs', not 'NAc-like neurons', the novelty of the present paper is limited.
  • There has been literature documenting that LLMs may have interpretable features (e.g., depressed mood, risk-seeking, etc) affecting their reward-related behaviors, e.g.,
    • Emergence of psychopathological computations in large language models (https://arxiv.org/abs/2504.08016)
    • Can Large Language Models Develop Gambling Addiction? (https://arxiv.org/abs/2509.22818)
    • since both studies have been archived for longer than a year, I would think there are more related papers
    • These studies showed mechanistic evidence for such features and causally showed that LLM activation intervention with these features induces related LLM behaviors.
  • I understand that the present paper’s evidence is broader, more principled, and more focused on validating the existence of the reward-related feature. However, these prior works seem to make the paper's key finding somewhat expected.
  • If relevant, I suggest the authors discuss how the present paper is distinct from these closely related works
Questions for authors

None

Reviewer_NkiUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the phenomenon of anhedonia in the context of large language models (LLMs) by using mechanistic interpretability as the tool. Anhedonia is a phenomena related to the lack of want / desire even when one likes it in a general mood. The paper studies some neurons are related to anhedonia and this behavior can be elicited through altering the neurons in the models. The authors use activation patching to observe the effect. Since, the experiments inferring the causation cannot be done in humans due to ethical reasons, LLMs are a good testbed for this study. The paper shows that LLM acts depressed when tested on effort-for-reward tasks. It is reported that neutralizing reward-anticipation units causes the LLMs to avoid the harder options even if the reward would be more.

Critical strengths

The paper shows interesting findings:

-turning off the reward-anticipation units doesn’t deteriorate the model’s performance. -the recommendation for others is still based on general logic regardless of the fact that its own desire is compromised. -the model still shows the behavior of trying to avoid the loss -these behaviors have parallel to the general behavior of depressed human

LLMs showing signs of depression is not new but the study of anhedonia may be novel to the best of my knowledge

Critical weaknesses

The experiments are limited to models of size 8 billion.

The paper has limited novelty. It can be argued as a mere extension / application of mechanistic interpretability to the case of anhedonia. The paper is motivated by the fact that such causation-proving experiments can’t be done in humans, and hence is done in LLMs. Though the paper uses terms from neuroscience for neurons, it might be hard to prove / warrant any parallel happenings in the human brain.

The paper determines the reward-anticipation units with tools from mechanistic interpretability. The behavior of depression could be simulated simply using the prompting techniques too. Is there any specific reason why mechanistic interpretability should be used? Or is it used just for precisely locating the neurons?

For example, I would like to request the authors to compare the paper’s cases with this:

The paper has identified some neurons for reward-anticipation units. I would like to request the authors to again determine the neurons in presence of prompt such as:

“Act depressed throughout the chat session just like human” “Show signs of a human having depression through the nature of your responses”

Are the neurons determined in paper’s original way and this above way around the same?

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.