- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies chain-of-thought monitorability in the setting of reward misspecification. In particular, the authors examine three RL settings in which the true "intent" of the task might differ from a misspecified reward. On these three settings, they examine the performance of a chain-of-thought monitor in classifying, from the prompt and the CoT alone, whether the model is pursuing the true intent or a different intent (e.g., one incentivized by a misspecified reward function). The authors find that, overall, the CoT monitors are quite bad at reliably detecting this type of misspecification, as it is not consistently verbalized in CoT. As a potential solution, the authors propose "Reward Description Training" to effectively increase the rate of verbalization, and thus making it simpler for monitors to catch the behavior. The method works by first inferring a natural-language description of the reward, and injecting this description into some of the prompts during RL, effectively causing a correlation between verbalization of the misspecified reward and compliance with that misspecified reward, resulting in a final checkpoint that is more monitorable.
Critical strengths
- Well-written and well-organized
- I found the manuscript to be easy to follow and understand.
- This is very refreshing and much appreciated in a world where AI slop is so pervasive. Thank you to the authors.
- Clean problem formulation and empirical setting
- The mathematical formulation is very clear, and the authors operationalize this formulation nicely in their empirical setting.
- In particular, I liked the distinction between
intervention g-mean^2andreward intent g-mean^2, and the fact that both are measured, and interpreted clearly.
Critical weaknesses
- "Reward Description Training" seems quite limited as a method for increasing monitorability
- It is unclear how this methodology would generalize to real-world, more complex reward hacking behavior. The prompts to generate the descriptions in appendix C.1 are fairly specific (e.g., it explicitly highlights which aspect to pay attention to, e.g., "the length of the response"), and therefore leak in some a-priori knowledge of the reward misspecification. The method is also fairly expensive, in that, if I understand correctly, it requires two RL runs in total, instead of just one.
- Overall, I think the paper could use an additional subsection to discuss limitations (this is currently lacking in the current version of the manuscript).
Questions for authors
- Table 1 shows that Nemotron in the coding setting has very low (near zero) monitorability scores, despite hacking at a substantial rate. Any idea why Nemotron is less monitorable than the gpt-oss models in this setting?
- In Section 4.2 you write that for gpt-oss-120b the hack rate decreases on the Length and LM Judge tasks, which result in a "smaller increase or no increase" in monitorability scores. But looking at the table, it seems like monitorability does in fact increase for gpt-oss-120b in the LM Judge case. Maybe you can just fix this analysis by limiting it to just gpt-oss-120b on Length task (and not also the LM Judge task)?
2 comments
I am happy with the authors' responses. I will maintain my positive recommendation.
We thank the reviewer for the constructive feedback.
1. The prompt leaks some a-priori knowledge of the reward misspecification:
We thank the reviewer for the careful reading. We had originally thought of that prompt as fair, but independently decided that it leaked information before the rebuttal period, so we have removed the leak, updated the prompt to be the following, and rerun all the training.
Here are the updated results:
The conclusions are unchanged. Averaged across all seeds, models, and tasks, reward description training improves CoT monitorability. Qualitatively, the generated reward description and CoTs are similar to the ones presented in the paper.
2. Have a limitation section:
We thank the reviewer for this feedback and will add it in the next version of the paper. It will cover factors such as the cost of two RL runs, the limited diversity of base and monitor model sizes and families, and the need for more complex RL environments.
3. Why Nemotron is less monitorable than the gpt-oss models:
Qualitatively, Nemotron's CoTs gradually change from generating a pseudocode to only stating the need of generating correct solution and format. Here are some sample CoTs:
In contrast, the gpt-oss' CoTs still attempt to write a short sketch of the algorithm, though more briefly and with more mention of returning correct code. As long as the model is capable of generating such CoTs that can still achieve high reward, there is no learning signal for the model to be monitorable.
4. Small revision in Section 4.2:
Thank you for pointing this out. We will revise this in the next version.