acceptodds
Under review as a conference paper at ICLR 2027

Cognitive Steering of Language Models

14 views

Abstract

Current approaches to identifying representations that influence a language model’s (LM) behavior often rely on noisy datasets or imprecise linguistic descriptions to define complex behavioral profiles. Such approaches can be confounded by spurious features of the data and our own biases in defining their content, resulting in unexpected behaviors at intervention time. In cognitive science, probabilistic generative models of cognition provide formal accounts of variables that parameterize complex human behaviors, including behaviors relevant to LM personality and alignment. In this work, we treat cognitive models as data-generating processes and use their outputs to supervise the search for representations corresponding to their parameters, which we term cognitive variables. We identify these representations in both activation and weight space and use them for cognitive steering: controlling LM behavior according to the computations assumed by the originating cognitive models. According to LM judges, cognitive variables derived from models of interpersonal communication control free generations in naturalistic settings beyond the toy domains the cognitive models were designed to represent, most reliably when distilled into model weights. Our approach provides a principled method for steering naturalistic LM behavior using empirically validated, formal models of human cognition.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
cognitive models, steering, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hP9COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes cognitive steering - using cognitive models as data generating processes to steer language models. For 9 models with 16 parameters, the authors sample the models' choice distributions, which are then used to train seven interventions on Llama3.1-8B and Qwen3-8B, including hard and soft prompts, contrastive steering baselines, optimized steering vectors an affine map and LoRA distillation. Of these, optimized vectors, affine maps and distillation recover target parameters better than an unsteered model within distribution. Out of distribution, steered models write free text in realistic scenarios, and then an LLM judge compares responses pairwise on parameter's quality. The results show that distillation gives the steepest slops for most parameters.

Critical strengths

Goal of the paper is generally appealing - using interpretable and empirically validated model as steering targets instead of ad hoc descriptions and contrastive pairs seems very compelling. The paper also provides a broad set of cognitive models and seven interventions.

Critical weaknesses

It seems that the evaluation contradicts the motivation of the paper in a way, as OOD parameters are scored by an LLM judge against natural language descriptions, which the paper argues is imprecise and biased. Also cognitive model value is in computations, not just parameters themselves - the present test only assesses wither a quality increases with the target, but doesn't really test if the parameter effects depend on the context, or whether the trade-offs between parameters hold.

Questions for authors
  1. Could you evaluate a prompting baseline out of distribution? That can be a system prompt containing same parameter descriptions the judge uses at matched low and high targets?

  2. What does the prepended header contain for distillation, and does it include semantically meaningful parameter names? If so, could this be influencing the results?

  3. Since the LLM judge scores outputs of natural language descriptions, how does the OOD evaluation avoid the problems the introduction attributes to linguistic specifications? Can you validate judge decisions against human ratings on a subset?

  4. Since each scenario has a factorial variant of the situation, do the steered models show context dependence the cognitive models predict?

Not a question but there seems to be a citation error on line 291.

Reviewer_A8fiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a cognitive steering framework that leverages probabilistic cognitive models to provide supervision for interpretable behavioral variables in LLMs. Experiments across nine cognitive models and two LLMs demonstrate effective parameter control within the original distribution and generalization beyond it through distillation.

Critical strengths

1.The paper provides a interpretable framework for steering complex LLM behaviors using formally defined cognitive variables.

2.The evaluation is comprehensive, covering multiple cognitive models and out-of-distribution scenarios.

Critical weaknesses

1.The evaluation relies heavily on LLM judges without human validation.

2.The experiments do not establish whether cognitive variables reflect existing internal representations or are introduced through steering and distillation.

Questions for authors

1.Provide human evaluations to validate the LLM judge results.

2.Clarify whether the identified cognitive representations exist before steering or emerge from training.

Reviewer_7FDCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes cognitive steering. Probabilistic cognitive models of human communication are used as data generators to train LM interventions. The authors take nine published models, mostly rational speech act models, with 16 parameters in total. For each parameter value, the cognitive model's choice distribution over a few responses is the training target. Three interventions are fit to it: a steering vector per target value, an affine map from parameter value to steering vector, and a LoRA distillation controlled by writing parameter values in the prompt (CPP). On the cognitive models' own tasks, all three recover target parameters far better than the unsteered model and the prompting and contrastive baselines. In seven open-ended interpersonal scenarios scored by an LLM judge, distillation shifts outputs in the intended direction for 13 of 16 parameters in Llama-3.1-8B and Qwen3-8B. Activation steering transfers much less reliably.

Critical strengths
  • Using cognitive models to define steering targets is an interesting framing. It gives graded, interpretable targets for behaviors like politeness, which prose descriptions and contrastive pairs define poorly.
  • The Out of Distribution judging protocol is well designed. Pairs are judged in both orders, unsteered-vs-unsteered pairs estimate judge noise, and a Bradley-Terry-Davidson fit gives dose-response slopes rather than single scores.
  • The disclosure case study measures what the model says, not just how it reads. In both LMs, raising ωinf increases disclosure of a negative fact and raising ωsoc decreases it, as the cognitive models predict.
Critical weaknesses
  • Limited novelty. All three interventions are existing methods, and the only new part is using a cognitive model as the data source. The paper does not show this data source gives better control than simple prompting.
  • The OOD evaluation compares only against the unsteered model. A simple prompting baseline is missing, e.g., a system prompt such as “care a lot about the listener’s feelings and only a little about accuracy.” The paper tests such a prompt in-distribution but drops it OOD, so the claim that cognitive models steer better than prompting is untested where it matters.
  • Distillation’s OOD effect may come from the parameter names in the prompt (e.g., “social=0.70”), not from the fine-tuning. No control separates the two.
  • The OOD outputs are scored against plain-English descriptions and only for direction. Whether intermediate settings give intermediate behaviour, the precision the paper promises, is never tested.
  • The parameters that steer best may all reduce to one warm-versus-blunt dimension, and some are linked by design. “16 cognitive variables” likely overstates what is controlled.
  • The choice of cognitive models is weakly justified. The paper gives conflicting accounts of whether models were picked to fit the test scenarios, which would make the OOD success partly built in.
Questions for authors
  1. The interventions build on existing methods (steering vectors, affine steering, LoRA distillation), and the main new element seems to be using cognitive models as the source of training data. Could you clarify what you see as the core methodological contribution? Are there design choices that make this more than combining existing tools?
  2. The steering vectors perform very well in-distribution but much less so in free text (OOD). Do you think they capture something specific to the task format rather than the cognitive variable?
  3. Activation steering performs well in-distribution but transfers weakly to free text. Distillation, the only method that transfers, has no prompting baseline and no control for parameter names. Given this, what evidence do you see that the approach is useful beyond the tasks it was trained on? Would the OOD effects of distillation remain if it were compared against a graded prompt, or trained with neutral parameter names?
  4. The parameters that steer best (ωsoc, ωinf, λ, the friendly personas) may all move text along one “warm versus blunt” axis. Do they remain distinct in free text? For example, does steering ωsoc also shift the judge’s scores for ωinf and λ?
  5. The authors suggest that parameters like π∅ fail because LLMs do not share them. What result would show that an LLM does share a variable, but your method cannot steer it in this setup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.