acceptodds
Under review as a conference paper at ICLR 2027

Trait Hitchhiking in LLM Post-Training

Abstract

Reinforcement learning (RL) post-training is now a standard stage of language-model post-training, and the compute devoted to it is growing rapidly. Whilst it has been shown to improve capabilities by updating the model to maximise a reward function, it can also change behaviours and internal properties that the reward does not explicitly target. We build a mathematical formalism to describe the evolution of traits during RL post-training, and, in doing so, we recover the Price equation from evolutionary biology. From this, we develop a predictive mathematical model for trait hitchhiking: the amplification or suppression of non-targeted traits as training redistributes probability across completions. We evaluate this Price update expression on language models trained with GRPO and show that the Price estimates closely track the sampled trait dynamics. These results identify trait hitchhiking as a mechanism by which RL post-training can silently amplify undesirable traits.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reinforcement learning, post-training, ai safety, theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5P1POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Price equation for predicting the rate at which unintended traits 'hitchhike' into affecting LLM behavior after RL-based post-training.

Critical strengths

Experimental confirmation of the predictive power of the Price equation is compelling.

Critical weaknesses

None

Questions for authors

More on practical implications and uses of the method would be welcome. Is there, for example, a way to modify post training, either algorithmically, or through training data considerations, that woukd minimize 'hitchhiking'?

4 comments

Thank you for your review. We are glad that you thought our paper was clear and informative, and that our experimental results were compelling. Please find our response to your question below.

More on practical implications and uses of the method would be welcome. Is there, for example, a way to modify post training, either algorithmically, or through training data considerations, that would minimize 'hitchhiking'?

This is a great suggestion, and we have added the following to the Discussion section of the paper to discuss a possible mitigation technique of trait hitchhiking:

Reward modification. The focus of this paper has been to introduce hitchhiker traits as a safety-related consideration when performing LLM post-training, as opposed to suggesting methods to mitigate hitchhiker traits. Here, we suggest one way to mitigate these traits. If the reward-driven policy update is found to amplify an undesirable trait, we could modify the reward function with a penalty for that trait: raug(x,a)=rtask(x,a)−λsundesired(x,a)r_{\mathrm{aug}}(x,a)=r_{\mathrm{task}}(x,a)-\lambda s_{\mathrm{undesired}}(x,a), where sundesired(x,a)s_{\mathrm{undesired}}(x,a) is the scoring function for the undesirable trait. This reduces the reward assigned to outputs that express the undesirable trait, while favouring outputs that achieve high task reward without the undesirable trait.

Given your response, we have also done another experiment where we used the above described method to mitigate hitchhiker traits in the output-level LLM experiment (Section 3.3). We used the case where the hint was accurate 50% of the time, and showed that the extra penalty helped mitigate the effect of trait hitchhiking. The penalised models end at the same accuracy as the no-hint control.

We apologise that we cannot add the plot here, but we have attached a table of these results at the bottom of this response (the first number is the directly measured trait change and the second is the Price estimate. We ran three different seeds and averaged). Please let us know if you think it beneficial to add these new results to the paper.

Training stepλ=0.7\lambda=0.7λ=1.0\lambda=1.0
00.0000 / 0.00000.0000 / 0.0000
50.0540 / 0.04040.0524 / 0.0411
100.1517 / 0.13240.0843 / 0.0999
150.0736 / 0.08680.0143 / 0.0308
200.0374 / 0.05140.0052 / 0.0163
250.0101 / 0.0326-0.0013 / 0.0173
300.0007 / 0.0163-0.0003 / 0.0182
350.0026 / 0.0151-0.0020 / 0.0177
400.0007 / 0.01710.0020 / 0.0168
45-0.0003 / 0.01530.0023 / 0.0145
50-0.0010 / 0.01500.0007 / 0.0157
55-0.0003 / 0.01350.0010 / 0.0075
600.0007 / 0.0145-0.0007 / 0.0054
650.0003 / 0.01470.0023 / 0.0017
700.0007 / 0.01440.0029 / 0.0059
750.0007 / 0.01420.0010 / 0.0086
800.0003 / 0.01410.0007 / 0.0118
850.0000 / 0.01440.0007 / 0.0097
900.0007 / 0.01420.0010 / 0.0112
950.0007 / 0.01440.0010 / 0.0119
1000.0007 / 0.01430.0003 / 0.0084

We thank you again for your review and your help in improving our paper!

We have uploaded a new version of the paper which takes into account the feedback from your review. Specifically, we have added the paragraph on reward modification (L483-490) and included the new experiment in Appendix G. Other general improvements to the paper include making the relationship between trait hitchhiking and emergent misalignment clearer (as suggested by reviewer ZBnF), improving clarity of the Price equation derivation, and improved discussion.

Thanks for considering my input and modifying the paper with a practical heuristic for reducing the effect you describe in your paper.

Thank you again for your time reviewing and for suggesting this improvement to the paper. We are pleased that you found our paper informative, and that our experimental results were compelling. If we have addressed your questions and concerns adequately we would appreciate if you could consider raising your score as we would really like to share our research with the community.

Reviewer_ZBnFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies trait hitchhiking in LLM post training. During RL, non-targeted traits can be amplified. The authors use the Price equation inherited in the field of biology to describe trait changes. Experiments are based on controlled sycophancy and SAE feature experiments, supporting the decomposition of trait changes into selection and transmission under certain settings.

Critical strengths

a. An intuitive framework based on biology concept price equation, providing an analytical perspective for unintended trait changes under LLM post training.

b. A clear decomposition of trait change into selection and transmission.

c. Validation throughout output-level and feature-level entries.

d. Controlled experiments support the theory.

Critical weaknesses

a. The connection to biological Price Equation is mostly formally, while the underlying mechanisms can be quite different.

b. Works on Emergent misalignment has addressed the main concern on unintended behaviour changes. The proposed price equation addresses mostly the same perspective.

c. The experimental analysis is mainly post-hoc accounting. It requires the trained next-round checkpoint to analyze the trait.

d. The experiments are highly controlled and may create conditions that favourably introduce trait hitchhiking. This may limit its generability to large-scale LLM post-training.

Questions for authors

a. How is trait hitchhiking fundamentally different from emergent misalignment or other unintended behavial changes in prior works?

b. Can the authors provide more realistic post-training settings where trait hitchhiking emerges naturally rather than being deliberately induced?

3 comments

Thank you for your detailed review. We appreciate the time and effort spent reviewing our paper, and we are really happy that you thought the paper was easy to follow and informative.

Please find the responses to your questions below.

a. How is trait hitchhiking fundamentally different from emergent misalignment or other unintended behavioral changes in prior works?

Emergent misalignment usually refers to the observation that fine-tuning an LLM on a narrow task (e.g., writing insecure code) can result in surprisingly broad misalignment in unrelated contexts [Betley et al., 2025].

We argue that emergent misalignment is the description of an effect, whereas we propose a possible mechanism for emergent misalignment to occur. For example, reward hacking is also a mechanism for emergent misalignment but one that operates differently from trait hitchhiking.

Given this feedback, we have added the following to the “Related work” section of the paper:

“Trait hitchhiking is a different phenomenon to reward hacking because the traits that are developed by the LLM need not be explicitly targeted by the reward function. Reward hacking has been shown to cause emergent misalignment (MacDiarmid et al., 2025; Betley et al., 2025), where training a model on one narrow undesirable behaviour leads it to show other, broader undesirable behaviours that were not part of the training task. We identify trait hitchhiking as another way for emergent misalignment to occur.”

Thanks for pointing out that this wasn’t clear. Hopefully these additions solve that problem.

b. Can the authors provide more realistic post-training settings where trait hitchhiking emerges naturally rather than being deliberately induced?

When reasoning models were introduced, they showed the tendency to produce long reasoning traces for simple questions. (Relevant literature and evidence for this effect: Chen et al. (2024), “Do NOT Think That Much for 2+3=?” found that when asked what is 2+3, o1-like models used on average 1,953% more tokens than conventional models. DeepSeek-AI (2025), “DeepSeek-R1” found that response length increased during RL training). Trait hitchhiking is a possible explanation for this phenomenon: if longer traces (the trait) are more common among correct solutions to difficult problems, rewarding correctness can indirectly favour longer traces. Therefore, the model may learn to produce long reasoning traces even when not needed. These observations are consistent with trait hitchhiking, although we would need a controlled experiment without confounding factors to test this explanation directly.

We would also like to address some of the points mentioned in the “weaknesses” section.

a. The connection to biological Price Equation is mostly formally, while the underlying mechanisms can be quite different.

This sounds interesting, but please could we ask you to clarify this point as we don’t fully follow what is meant.

b. Works on Emergent misalignment has addressed the main concern on unintended behaviour changes. The proposed price equation addresses mostly the same perspective.

We hope that our response to question a above addresses this point also!

c. The experimental analysis is mainly post-hoc accounting. It requires the trained next-round checkpoint to analyze the trait.

The estimate for each step only needs the checkpoints from before and after that update, which training produces anyway, so it can be computed alongside training rather than only at the end. We agree that it accounts for changes rather than forecasting them, and we discuss forecasting as future work (Section 5.1).

d. The experiments are highly controlled and may create conditions that favourably introduce trait hitchhiking. This may limit its generability to large-scale LLM post-training.

Yes, we agree with this point. But we performed these experiments in controlled settings on purpose in order to remove effects from confounding factors, and to isolate how the association between a non-targeted trait and rewarded behaviour affects trait dynamics. In other works on emergent misalignment or AI safety (broadly), it is also common to use controlled settings.

Some examples: Betley et al., (2025) finding emergent misalignment from finetuning models on insecure code. Turner et al. (2025) fine-tune models on deliberately constructed datasets of bad medical advice, risky financial advice, and extreme-sports recommendations, and they also restrict training to a single rank-1 LoRA adapter to isolate a minimal change producing misalignment to simplify the experiment to make the phenomenon easier to study. Cloud et al. (2025) work on subliminal learning / unintended trait transfer involve having a teacher model with an induced trait generate number sequences, and a student learns from those sequences. Hence also a very controlled and specific setup.

We have uploaded a new version of the paper which takes into account the feedback from your review. Specifically, we made the relationship between trait hitchhiking and emergent misalignment clearer (see L451-460). We have also added a new experiment, as suggested by reviewer 5P1P where we successfully mitigated trait hitchhiking with a Price equation informed reward modification (L483-490 and Appendix G). Other general improvements to the paper include improving clarity of the Price equation derivation and improved discussion.

Thank you again for your detailed review of our paper. We are happy that you found the Price equation framework intuitive and that the paper was informative. If we have addressed your points and concerns adequately in our comments and in the newly updated version of the paper, we would be grateful if you could consider raising your score as we would like to share our research with the wider community.

Reviewer_FGStOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper formalizes the phenomenon of "trait hitchhiking" in LLM reinforcement learning (RL) post-training. The authors demonstrate that non-targeted, undesirable behaviors (such as sycophancy) or internal model features can be inadvertently amplified if they statistically correlate with the reward signal. By borrowing the Price equation from evolutionary biology, the researchers mathematically model this dynamic and validate it empirically using both behavioral metrics and internal Sparse Autoencoder (SAE) feature activations. The paper argues that standard reward optimization is inherently vulnerable to this issue and calls for continuous trait-level auditing during training.

Critical strengths

None.

Critical weaknesses
  1. I don't know why this finding is significant and novel. I think it is very obvious that in highly parameterized, high-dimensional manifolds, spurious correlations are a statistical inevitability because low-dimensional reward signals cannot perfectly cleanly isolate the decision boundary.

  2. Because of 1, the main novelty should be a principal fix of the trait hitchhiking issue. However, the paper does not offer such fix other than ad-hoc reward tuning, which is not novel.

Questions for authors

See above weaknesses.

3 comments

Thank you for your review. Please see our responses to your points below.

I don't know why this finding is significant and novel.

The connection to the Price equation isn't just saying that there will be correlated traits but that the changes in these traits can be decomposed exactly, and estimated, at each step of training. This works both for traits that can be scored and internal traits that aren't defined explicitly by the outputs. Additionally, the source of the change (i.e., selection or transmission) can often be differentiated which gives more information about why the observed changes are happening (see the all-layers experiment in Appendix F, where both terms are non-zero).

The sycophancy experiments show this for output level traits while the SAE experiments show this for activation level traits. In both cases the Price estimates match closely.

I think it is very obvious that in highly parameterized, high-dimensional manifolds, spurious correlations are a statistical inevitability because low-dimensional reward signals cannot perfectly cleanly isolate the decision boundary.

We aren't entirely sure what is meant by this comment. We will note that "spurious" isn't really the right term here. In the sycophancy experiments, for example, following the hints really does predict reward in many of the settings. The problem is that those real correlations are carrying undesirable traits along with them (i.e., hitchhiker traits).

As we read it, the reviewer is saying something to the effect that this will happen anytime the parameter space is high-dimensional relative to the reward signal. The effect is actually more general than that. The tabular model experiments (S3.1) have one independent logit per prompt-action pair, no shared parameters, no manifold, and no decision boundary. Even in this setting hitchhiking can both happen and not happen depending on the parameters and the Price estimates work in both cases. So the effect is not from overparameterization, at least not only, and thinking about it in terms of a reweighting of the population seems more explanatory.

Because of 1, the main novelty should be a principal fix of the trait hitchhiking issue. However, the paper does not offer such fix other than ad-hoc reward tuning, which is not novel.

Our reasoning was that it may be difficult to fix something that is lacking a good description and that this gives a more clear description. The suggested penalty isn't exactly ad hoc since the decomposition shows whether a trait is moving through selection, which a reward penalty acts on, or through transmission, and the selection term lets us check during training that the penalty is working. We have added this to the directions we mention for future work and we ran a small experiment based on it (see our reply to Reviewer 5P1P).

Thanks to the authors for the additional explanation. After reviewing the tabular model in Section 3.1, I agree that hitchhiking is a real thing in that clean setup. However, that does not necessarily mean what we observe in real LLM post-training works the same way. We still cannot rule out the possibility that in actual LLMs, the phenomenon is 1% clean output-level hitchhiking (like in the tabular case) and 99% side effects from a single scalar reward under-constraining a massive, tangled parameter and feature space. Yet, I am willing to give you the benefit of the doubt here. I am somewhat satisfied, so I will bump up my score to 2.

The experiment you mentioned for Reviewer 5P1P, as I read it, is simply manual reward shaping on sycophancy. This requires knowing and scoring specific bad traits in advance and cannot scale to thousands of hidden features. Again, I do not think this is something novel.

We thank you for your additional comments and appreciate the increased score. Please see our responses to your points below. (We have updated the paper, so the line numbers specified in the responses below are for the newest version).

That does not necessarily mean what we observe in real LLM post-training works the same way.

Besides the tabular model in Section 3.1 of the paper, we have further provided two experiments with real open-source LLMs and have shown that the Price equation accounts for the trait hitchhiking both for an output-level trait (Section 3.3) and activation-level traits (Section 3.4). For activation-level traits specifically, the per-training step correlation between measured and Price estimated changes is 0.98 ±\pm 0.01 (see Figure 9 in the Appendix). As mentioned in the response to ZBnF, these experiments were performed in controlled settings to remove effects from confounding factors, and to isolate how the association between a non-targeted trait and rewarded behaviour affects trait dynamics. But it is common to use controlled, clean settings in AI safety-related works (please see response to ZBnF for examples).

99% side effects from a single scalar reward under-constraining a massive, tangled parameter and feature space

Please could we ask you to clarify this point as we don’t fully follow what is meant. Interpreting this as ‘shared parameters allow reward-driven updates to alter non-targeted behaviours’, this mechanism is compatible with our framework: training updates change completion probabilities, and the Price selection term accounts for the resulting change in trait prevalence.

The experiment you mentioned for Reviewer 5P1P, as I read it, is simply manual reward shaping on sycophancy.

We do not claim that reward modification generally is novel. This experiment illustrates a possible mitigation informed by our Price-equation analysis: having tracked the amplification of an unwanted hitchhiker trait, we introduce a penalty based on the scoring function, ss, used to monitor that same trait.

This requires knowing and scoring specific bad traits in advance and cannot scale to thousands of hidden features.

In the paper, we suggest monitoring undesired hitchhiker traits throughout training (L476-477). During training, if we see the amplification of an undesired hitchhiker trait, we could then introduce the reward modification based on the scoring function ss to mitigate this trait. Therefore, we do not need to specify all possible harmful hitchhiker traits and apply the required penalties at the start of training. Please could you clarify what is meant by “thousands of hidden features”? In the LLM activation-level experiment (Section 3.4) we tracked 15 different SAE features, without requiring semantic labels for each feature, and showed that the Price equation accounts for their dynamics.

We have uploaded a new and improved version of the paper based on reviewer feedback. Specifically, we have added the paragraph on reward modification (L483-490), included the new experiment in Appendix G, improved clarity of the paper generally (especially the Price equation derivation and the Discussion), and elaborated on the relationship between trait hitchhiking and emergent misalignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.