acceptodds
Withdrawn from ICLR 2027

Developmental Cognitive Interpretability: Modelling Value Learning in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Narrowly fine-tuning an LLM to express one value can generalise to others it was not directly trained on, making the full effects of complex alignment post-training methods hard to predict. Understanding these effects is becoming increasingly important to ensure correct value alignment of LLMs as they become more capable and are deployed into a wider variety of situations, even during training. To aid this understanding, we propose developmental cognitive interpretability (DCI), a framework that seeks to predict how training influences an agent's behaviour, without having to carry out that training. It models an agent's behaviour as arising from some latent state, according to a cognitive model, with a developmental rule determining how that state changes during training. We study this framework by training Qwen3.5-9B with DPO to express different behavioural values, and then evaluating the effect this has on its values more broadly. We find that changes in behavioural value expression can be compactly expressed: a cognitive model assigning a score to each value predicts held-out preferences, and changes in these scores are low-dimensional. Furthermore, a developmental cognitive model fitted to a population of agents each trained to express a single value can predict the expressed values of agents with more complex training pipelines. Overall, our results show that compact models of the latent dynamics of behavioural value generalisation in LLMs can provide accurate predictions, even out of distribution. Such models offer a way to investigate and potentially anticipate the broader behavioural effects of narrow alignment training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Developmental cognitive interpretability, large language models, value learning, value generalisation, cognitive modelling, AI alignment, generalisation, out-of-distribution prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jsz9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyzes how one can predict the alignment towards a value during post-training. The paper introduces developmental cognitive interpretability (DCI), which treats behaviour as a latent state through a cognitive model. Training Qwen3.5-9B with DPO toward individual values, the authors find that a per-value score model predicts held-out preferences, and training-induced score changes are low-dimensional. A developmental model fitted only to single-value training also predicts values after multi-stage pipelines. The results suggest that models of value-generalisation dynamics can predict the effects of narrow alignment training.

Critical strengths
  1. The motivation is clear and highly relevant. The effects of value alignment are currently quite opaque.
  2. The general findings of this work, such as that training-induced value shifts are low-dimensional, are interesting and can inform model post-training/value alignment.
Critical weaknesses
  1. The paper is not self-contained and is hard to follow. The core parts of the framework (cognitive model, cognitive state, developmental rule, agent) are introduced informally in Section 2, without formal definitions or examples. “Agent” is defined only in Section 3, and the formal definitions appear only in Appendices B.1 and C.1. The main text does not describe how value scores are estimated, and it refers to Hua et al. (2026), with the fitting objective given only in Appendix B.3. Section 4.3 introduces w, ϕ, C, and D in passing, without saying how the latent-Gaussian model is fitted. Figure 1 does not clarify the pipeline, and methodology is interleaved with results throughout Sections 4–6. This is important because the paper’s primary contribution is its framework.

  2. The measured values are not validated beyond an LLM judge. All preference rates come from Haiku 4.5 grades on a 0–6 rubric, without examples for every grade. The scenario generator comes from the same model family as the judge. There is no human evaluation. Without evidence that these scores track how humans perceive the responses’ value expression, it is unclear whether DCI models value learning or judge behaviour. This limits the significance of the paper’s alignment-relevant conclusions. The concern is also about the narrow scope of a single LLM with one training run per pipeline.

Questions for authors
  1. How is a cognitive model shared across agents? Motivations can lead to different behaviours for different agents, unless I don’t understand your definition of an agent (which is not given in the text until this point).
  2. Have you experimented with a narrower rubric scale, for example, 0–2 or a binary choice? The examples in the fig are relatively opinionated but get scores 2 and 4. What do responses graded 1 or 5 look like?
Reviewer_eYefOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether we can predict how an LLM’s answers will change after several rounds of fine-tuning. The authors first train separate versions of Qwen toward individual values, such as honesty or privacy, and measure how their answers change. They use these observations to build a small mathematical predictor. They then test whether it can predict the behavior of a model trained toward one value and subsequently another, without learning from that combined training run. The predictions improve when they account for both stages rather than only the last one. Simply adding the changes predicted for each stage separately performs about as well as their sequential prediction method.

Critical strengths
  1. The paper studies a useful question: can we predict the effects of combining training stages from what each stage does separately? The results show that accounting for both training stages predicts the final model’s behavior better than using information from the second stage alone.

  2. The authors compare their approach with simpler alternatives and clearly report where it falls short. In particular, they show that simple addition works similarly well and acknowledge that their predictor misses an observed effect of training order.

Critical weaknesses
  1. The benefit of sequential modelling is unclear: 1.1. Similar performance on simple addition. The additive baseline combines the changes predicted from A-only and B-only training to predict A → B behavior. It performs about as well as the sequential method.

1.2 The source of the predictability is unclear. The model may learn to combine A-related and B-related behavior into answers that satisfy aspects of both values. If so, the success of additive prediction could reflect this relatively simple combination, rather than support the proposed account of value development. Response-level analysis would help distinguish these explanati

  1. The possible combination of behaviors in 1.2 may be specific to Qwen or DPO. If the model learns to combine A-related and B-related behavior into answers that satisfy aspects of both values, the resulting predictability may depend on this particular model or training procedure. All two-stage experiments use Qwen3.5-9B with DPO, so the paper does not establish whether this behavior—and the success of predicting it—extends to other models or training methods
Questions for authors

Could you clarify the concerns raised in the weaknesses above, particularly the possible explanation for additive predictability and whether it generalizes beyond Qwen3.5-9B and DPO?

Reviewer_CmaBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Developmental Cognitive Interpretability (DCI), a framework for predicting how fine-tuning changes the behavioral values expressed by language models without performing the target training. It combines a cognitive model that represents an LLM’s preferences through latent value scores with a developmental rule describing how these scores evolve during training. Evaluating the approach using Qwen3.5-9B, fine-tuned with DPO and SFT toward twelve behavioral values. It is first shown that compact value representations predict held-out pairwise preferences and that training-induced changes exhibit low-dimensional structure then fit developmental models to single-stage training trajectories and evaluate their ability to predict unseen combinations of two training stages. The Gaussian latent policy gradient model achieves prediction errors close to models trained directly on two-stage histories and outperforms memoryless baselines. However, simple additive predictions perform comparably, suggesting that much of the observed generalization may reflect approximately separable training effects.

Critical strengths
  1. The strongest contribution I feel is the demonstration that developmental models fitted only on single-stage DPO trajectories can predict behavioral preferences after previously unseen two-stage training sequences. The Gaussian latent policy gradient model achieves prediction errors close to a reference model fitted directly on other two-stage histories and consistently improves over memoryless predictions. This provides meaningful evidence that training-induced behavioral changes can be modeled in a way that transfers across training stages, supporting the paper's central motivation.

  2. The paper shows that pairwise value preferences can be approximated using a small number of latent scores and that training-induced changes exhibit substantial low-dimensional structure, with the first three singular components explaining 86.8% of the squared changes. This provides empirical justification for the proposed framework and offers a useful way to study how optimizing one behavioral value influences others without analyzing the full parameter space of the LLM.

Critical weaknesses
  1. Although the proposed models predict unseen two-stage training histories reasonably well, the simple additive baseline achieves comparable average prediction error. More importantly, the compositional models fail to reproduce the observed direction of the average training-order effect, predicting slightly stronger expression of values trained second when the experiments show the opposite. This raises concerns about whether the learned developmental rules capture meaningful interactions between successive training stages or primarily approximate additive changes in behavioral preferences. Since modeling training dynamics is the paper's central contribution, I find this limitation consequential. Can the authors discuss this limitations?

  2. The experiments are limited to a single LLM (Qwen3.5-9B), twelve selected values, synthetic evaluation scenarios, and relatively constrained LoRA-based training pipelines, with only one training run per pipeline. While the two-stage experiments provide useful evidence of generalization across training histories, they do not establish whether the framework remains reliable across different architectures, training seeds, or more realistic alignment procedures. Furthermore, the reliance on LLM-generated scenarios and LLM-based grading introduces uncertainty about how faithfully the measured preference scores reflect broader behavioral values. These limitations weaken the generality of the paper's conclusions, although they do not negate its promising empirical findings. Can you provide additional experiments ?

Questions for authors

Please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.