- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes the Price equation for predicting the rate at which unintended traits 'hitchhike' into affecting LLM behavior after RL-based post-training.
Critical strengths
Experimental confirmation of the predictive power of the Price equation is compelling.
Critical weaknesses
None
Questions for authors
More on practical implications and uses of the method would be welcome. Is there, for example, a way to modify post training, either algorithmically, or through training data considerations, that woukd minimize 'hitchhiking'?
4 comments
We have uploaded a new version of the paper which takes into account the feedback from your review. Specifically, we have added the paragraph on reward modification (L483-490) and included the new experiment in Appendix G. Other general improvements to the paper include making the relationship between trait hitchhiking and emergent misalignment clearer (as suggested by reviewer ZBnF), improving clarity of the Price equation derivation, and improved discussion.
Thanks for considering my input and modifying the paper with a practical heuristic for reducing the effect you describe in your paper.
Thank you again for your time reviewing and for suggesting this improvement to the paper. We are pleased that you found our paper informative, and that our experimental results were compelling. If we have addressed your questions and concerns adequately we would appreciate if you could consider raising your score as we would really like to share our research with the community.
Thank you for your review. We are glad that you thought our paper was clear and informative, and that our experimental results were compelling. Please find our response to your question below.
This is a great suggestion, and we have added the following to the Discussion section of the paper to discuss a possible mitigation technique of trait hitchhiking:
Given your response, we have also done another experiment where we used the above described method to mitigate hitchhiker traits in the output-level LLM experiment (Section 3.3). We used the case where the hint was accurate 50% of the time, and showed that the extra penalty helped mitigate the effect of trait hitchhiking. The penalised models end at the same accuracy as the no-hint control.
We apologise that we cannot add the plot here, but we have attached a table of these results at the bottom of this response (the first number is the directly measured trait change and the second is the Price estimate. We ran three different seeds and averaged). Please let us know if you think it beneficial to add these new results to the paper.
We thank you again for your review and your help in improving our paper!