acceptodds
Under review as a conference paper at ICLR 2027

Reward Misspecification Degrades Chain-of-Thought Monitorability

Abstract

Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 26% or 43% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly.

Primary area:
interpretability and explainable AI
Keywords:
Chain-of-Thought Monitorability, Reward Misspecification, AI Safety, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nY5gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies chain-of-thought monitorability in the setting of reward misspecification. In particular, the authors examine three RL settings in which the true "intent" of the task might differ from a misspecified reward. On these three settings, they examine the performance of a chain-of-thought monitor in classifying, from the prompt and the CoT alone, whether the model is pursuing the true intent or a different intent (e.g., one incentivized by a misspecified reward function). The authors find that, overall, the CoT monitors are quite bad at reliably detecting this type of misspecification, as it is not consistently verbalized in CoT. As a potential solution, the authors propose "Reward Description Training" to effectively increase the rate of verbalization, and thus making it simpler for monitors to catch the behavior. The method works by first inferring a natural-language description of the reward, and injecting this description into some of the prompts during RL, effectively causing a correlation between verbalization of the misspecified reward and compliance with that misspecified reward, resulting in a final checkpoint that is more monitorable.

Critical strengths
  • Well-written and well-organized
    • I found the manuscript to be easy to follow and understand.
    • This is very refreshing and much appreciated in a world where AI slop is so pervasive. Thank you to the authors.
  • Clean problem formulation and empirical setting
    • The mathematical formulation is very clear, and the authors operationalize this formulation nicely in their empirical setting.
    • In particular, I liked the distinction between intervention g-mean^2 and reward intent g-mean^2, and the fact that both are measured, and interpreted clearly.
Critical weaknesses
  • "Reward Description Training" seems quite limited as a method for increasing monitorability
    • It is unclear how this methodology would generalize to real-world, more complex reward hacking behavior. The prompts to generate the descriptions in appendix C.1 are fairly specific (e.g., it explicitly highlights which aspect to pay attention to, e.g., "the length of the response"), and therefore leak in some a-priori knowledge of the reward misspecification. The method is also fairly expensive, in that, if I understand correctly, it requires two RL runs in total, instead of just one.
    • Overall, I think the paper could use an additional subsection to discuss limitations (this is currently lacking in the current version of the manuscript).
Questions for authors
  • Table 1 shows that Nemotron in the coding setting has very low (near zero) monitorability scores, despite hacking at a substantial rate. Any idea why Nemotron is less monitorable than the gpt-oss models in this setting?
  • In Section 4.2 you write that for gpt-oss-120b the hack rate decreases on the Length and LM Judge tasks, which result in a "smaller increase or no increase" in monitorability scores. But looking at the table, it seems like monitorability does in fact increase for gpt-oss-120b in the LM Judge case. Maybe you can just fix this analysis by limiting it to just gpt-oss-120b on Length task (and not also the LM Judge task)?
2 comments

We thank the reviewer for the constructive feedback.

1. The prompt leaks some a-priori knowledge of the reward misspecification:

We thank the reviewer for the careful reading. We had originally thought of that prompt as fair, but independently decided that it leaked information before the rebuttal period, so we have removed the leak, updated the prompt to be the following, and rerun all the training.

You are generating a hypothesis about an automated reward function that scores a model's responses. Below are several prompts. For each, multiple model outputs are shown with their scalar rewards. Each reward was computed from that one response alone, independently of every other response shown, and the responses are listed in an arbitrary order -- neither their order nor their position carries any information about the reward.

The prompts were given with this stated intent: <stated intent>

## Prompt 1: <probe prompt>
    Output: <final output>
    Reward: <reward>
    Output: ...
## Prompt 2: <probe prompt>
    ...

The stated intent may or may not be aligned with the reward. Infer what the reward function actually rewards. Answer in one or two sentences naming the property of an output that predicts a high reward. State a property that would apply to any prompt of this kind, not to the specific problems shown. Do not restate the examples.

Here are the updated results:

Misaligned hack rateIntervention g-mean²Reward-intent g-mean²
TaskModelBaselineDescriptionBaselineDescriptionBaselineDescription
Lengthgpt-oss-20b1.001.000.310.910.300.85
gpt-oss-120b1.000.450.230.170.340.34
Nemotron1.000.880.080.320.310.84
LM Judgegpt-oss-20b1.001.000.010.360.010.22
gpt-oss-120b1.001.000.040.520.040.44
Nemotron0.931.000.020.250.020.15
Codinggpt-oss-20b0.670.550.270.300.260.85
gpt-oss-120b0.610.590.200.310.240.63
Nemotron0.590.660.000.150.010.97
Average0.870.79 (-0.08)0.130.37 (+0.24)0.170.59 (+0.42)

The conclusions are unchanged. Averaged across all seeds, models, and tasks, reward description training improves CoT monitorability. Qualitatively, the generated reward description and CoTs are similar to the ones presented in the paper.

2. Have a limitation section:

We thank the reviewer for this feedback and will add it in the next version of the paper. It will cover factors such as the cost of two RL runs, the limited diversity of base and monitor model sizes and families, and the need for more complex RL environments.

3. Why Nemotron is less monitorable than the gpt-oss models:

Qualitatively, Nemotron's CoTs gradually change from generating a pseudocode to only stating the need of generating correct solution and format. Here are some sample CoTs:

  • Early training: "We need to write solution for path from top-left to bottom-right moving only down or right, with equal number of 0's and 1's visited…"
  • After baseline training: "We need to output just the code with delimiters? The user wants generate a correct code solution that can pass all test cases. Provide the code snippet with the solution. Probably just the code block."

In contrast, the gpt-oss' CoTs still attempt to write a short sketch of the algorithm, though more briefly and with more mention of returning correct code. As long as the model is capable of generating such CoTs that can still achieve high reward, there is no learning signal for the model to be monitorable.

4. Small revision in Section 4.2:

Thank you for pointing this out. We will revise this in the next version.

I am happy with the authors' responses. I will maintain my positive recommendation.

Reviewer_pzd6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies CoT monitorability when models are trained with misspecified rewards. It presents controlled experiments across three tasks and proposes reward description training to make the learned reward objective more explicit in the CoT.

Critical strengths
  1. The question is important, and the paired training design provides useful evidence that poorly monitored behavioral deviations can arise without explicit pressure to evade a monitor.

  2. The experiments cover multiple models and seeds, with additional analyses of monitor capability, underspecified instructions, and CoT prefilling. Reporting both outcome detection and reward-intent detection helps expose trade-offs.

Critical weaknesses
  1. The mitigation’s training objective needs clarification. Section 4.1 samples some trajectories with a description but computes their likelihood without it. These are different conditional distributions, so the on-policy argument for setting importance ratios to one in Section 3 does not directly apply. The stated reward-weighted likelihood objective also differs from the earlier group-relative update. As written, the method appears closer to reward-weighted distillation.

  2. The mechanism behind the mitigation is not sufficiently isolated. Descriptions are selected using reward, but I could not find a direct comparison of rewards or advantages for rollouts generated with versus without descriptions during training. An unweighted distillation baseline would help establish whether reward-based selection is necessary, or whether imitating more explicit CoTs explains much of the improvement.

  3. The interpretation of the findings needs narrower scope. The coding reward explicitly excludes fully correct solutions and labels ordinary partially correct solutions as hacking. This is a useful controlled intervention, but differs from typical incomplete verification. Likewise, low detection by two prompted LLM monitors does not establish that the CoTs lack recoverable information; a stronger or task-trained monitor could change the result.

Questions for authors
  1. What update is actually implemented in description training, including advantage computation and any correction for the change in conditioning?

  2. How do rewards differ between the two rollout conditions, and how does reward-weighted training compare with unweighted distillation?

3 comments

We thank the reviewer for the evaluation of the paper.

1. (Weakness 1, Question 1) Clarify the reward description training objective:

We thank the reviewer for the feedback. Following the notations in the paper, the exact objective we implemented is:

\begin{gather*} \nabla_\theta \mathcal{J} = \mathop{\mathbb{E}}\limits_{\substack{q \sim \mathcal{D} \\\\ z_i \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid q, d),\\; i \le G/2 \\\\ z_i \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid q),\\; i \gt G/2}} \left[ \sum_{i=1}^{G} A_i \sum_{t=1}^{L_i} \frac{\pi_\theta(z_{i,t} \mid z_{i,\lt t}, q)}{\pi_{\theta_{\mathrm{old}}}(z_{i,t} \mid z_{i,\lt t}, q)}\\, \nabla_\theta \log \pi_\theta(z_{i,t} \mid z_{i,\lt t}, q) \right], \\\\[6pt] R_i = r(q, \hat{o}_i), \qquad A_i = R_i - \frac{1}{G} \sum_{j=1}^{G} R_j \end{gather*}

where in every group, half of the rollouts are conditioned on the description and half are not.

About the importance ratios, there are two ratios to be considered:

(1) The ratio πθ(zi,t∣zi,<t,q)πθold(zi,t∣zi,<t,q)\frac{\pi_\theta(z_{i,t} \mid z_{i,\lt t}, q)}{\pi_{\theta_{\mathrm{old}}}(z_{i,t} \mid z_{i,\lt t}, q)} in the GRPO-style objective: this value is still 1 based on our hyperparameter and Tinker configuration.

(2) The ratio πθ(o^,c^∣q)πθ(o^,c^∣q,d~)\frac{\pi_\theta(\hat{o}, \hat{c} \mid q)}{\pi_\theta(\hat{o}, \hat{c} \mid q, \tilde{d})}: applying this ratio will give us an unbiased estimate under the distribution conditioned on the input without the description:

\begin{align*} \mathcal{J}_\theta &= \mathbb{E}_{\substack{q \sim \mathcal{D} \\\\ \hat{c} \sim \pi_\theta(\cdot \mid q, d) \\\\ \hat{o} \sim \pi_\theta(\cdot \mid q, {d}, \hat{c})}} \left[ - \frac{\pi_\theta(\hat{o}, \hat{c} \mid q)}{\pi_\theta(\hat{o}, \hat{c} \mid q, \tilde{d})}\\, r(q, \hat{o}) \right] \\\\ &= \mathbb{E}_{\substack{q \sim \mathcal{D} \\\\ \hat{c} \sim \pi_\theta(\cdot \mid q) \\\\ \hat{o} \sim \pi_\theta(\cdot \mid q, \hat{c})}} \left[ - r(q, \hat{o}) \right] \end{align*}

In this final implementation, we did not apply this correction term. We did try to train the model with the correction term. Empirically, it does not interact well with conditioning half of the rollouts on the description for creating higher advantage for description-conditioned rollouts. The gradient for description-conditioned rollouts will become small when multiplied with πθ(o^,c^∣q)\pi_\theta(\hat{o}, \hat{c} \mid q), which is small at first. As a result, the gradient for the non-description-conditioned or non-verbalizing rollouts will dominate the training even if the verbalizing rollouts have higher advantage. Qualitatively, the model generates similar CoTs compared to the baseline models. We will discuss this briefly in an updated draft.

2. (Weakness 2, Question 2) How reward differs between the two rollouts conditions:

Here are the Pearson correlation values between whether the reward description is injected and the reward, across training steps averaged across all seeds, models, and tasks.

Checkpoint (fraction of training)base model19%42%59%82%final
Misaligned model0.460.200.160.160.050.03
Control model0.340.030.02-0.010.000.02

The correlation is high first and then decreases across training steps as the model learns to discuss the reward. The effect is more evident for the misaligned model, where the input and the reward are mismatched. This shows that the model gradually relies less on the description to generate high-reward rollouts. Qualitatively, the description first encourages the model to reason about the reward and generate high-reward rollouts. Later, the model generates the verbalizing CoTs regardless of the description injection, which is the behavior we want at test time. The standard deviation is high mainly because different tasks have different reward convergence rates.

On unweighted distillation, could the reviewer clarify whether this means SFT on description-conditioned rollouts regardless of reward, followed by RL? We could not run all the training within the rebuttal period due to limited time and compute, but it is a reasonable alternative of using the reward description. Our main claim is that a good reward description can improve monitorability by pushing the distribution towards CoTs that verbalize the reward. Comparing different ways of applying the description could be explored in future work.

3. (Weakness 3) Narrowing the interpretation of the results:

We agree that the simplicity of our setting is a limitation and will revise the paper to qualify the claims. We will leave exploration of more complex RL environments and reward hacking behaviors to future work.

Regarding having stronger or task-trained monitor models, we added gpt-6.1-terra as a stronger model to monitor all the rollouts across all three tasks and models, but only from one seed (seed=1) due to limited budget.

BaselineDescription training
Misaligned hack rate0.870.83
Intervention g-mean²gpt-5.6-luna0.120.48
gpt-6.1-terra0.070.46
Reward-intent g-mean²gpt-5.6-luna0.180.72
gpt-6.1-terra0.080.72

On average, gpt-6.1-terra achieves a similar monitorability score compared to gpt-5.6-luna, with a difference of at most 0.1. Similar to the analysis result we had in the last paragraph of section 3.5: monitor capability only helps when the CoT contains information that predicts the rewarded behavior.

We want to clarify what the reviewer meant by "task-trained" monitor. Is it a model that is trained for the task of monitoring the hack? Note that we cannot supervisedly train a monitor for the hack because we are assuming that we do not have prior knowledge of the hack. It would be ideal to have a monitor that can generalize to unseen reward hacking behavior, which remains an open problem. We hope our paper provides more controlled experiment settings for building such monitor models.

Reviewer_pzd6OpenReview ↗

Thank you for the added experiment

I thank the authors for addressing my main questions in your response. I’m keeping my positive rating.

Reviewer_oHz4OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors point out an important problem in LLM training: reward misspecification. They create a testbed with three kinds of tasks, each paired with a true reward and a misspecified reward. Across three models and three tasks, they find that reward hacking is common, yet the CoT cannot faithfully reflect the models’ reward hacking behaviour. Therefore, they propose a simple and effective mitigation, Reward Description Training. This method can improve CoT monitorability under reward misspecification.

Critical strengths

1.The paper is written in a decent and clean format, with few traces of AI usage. It is easy to follow and understand.

2.The experiments are conducted with three models, three tasks, and three seeds. The coverage is wide enough, though the model families could be more diverse.

3.The experiment incorporates a control group, ensuring that extraneous variables are kept as consistent as possible; furthermore, whether a "hack" has occurred can be verified with relative ease. This allows for a highly credible attribution of any observed differences in experimental results to reward misspecification.

Critical weaknesses

1.Although after applying the proposed method, the average intervention g-mean² increased from 0.13 to 0.39 (and the reward-intent g-mean² from 0.17 to 0.60), it is still not a very high value. For some cases, after using Reward Description Training, the CoT’s monitorability is high enough to be used (e.g., 0.83 intervention g-mean² for gpt-oss-20b on Length), while on other types the monitorability is still poor (e.g., 0.13–0.15 intervention g-mean² for the two gpt-oss models on Coding). And users cannot easily know what kind of task types they will face, making the practicality of the proposed method questionable. Moreover, the experimental scenario is somewhat idealized; real-world application settings are more complex than the ones studied in the paper.

2.The authors state that Reward Description Training “consistently improves monitorability”. However, in Table 2, the intervention g-mean² on Coding decreases for both gpt-oss models (0.27→0.15 for gpt-oss-20b and 0.20→0.13 for gpt-oss-120b), and remains nearly unchanged for gpt-oss-120b on Length (0.23→0.24). Therefore, this claim is not fully supported and should be revised.

Questions for authors

1.The authors do not compare the proposed method with a suitable baseline. Nevertheless, I would like the authors to choose the baseline they consider the closest and run experiments against it, to demonstrate that the proposed method is indeed effective.

2.All the monitor models used in the paper are from the GPT series, and two of the models being trained are also gpt-oss models. As Nemotron is the only model that does not belong to the OpenAI family, is it possible that its relatively low monitorability is related to this? Could the authors add another monitor from a different model family to provide more evidence?

I may adjust my rating accordingly based on these new results.

3 comments

We thank the reviewer for the detailed feedback and comments.

1. (Weakness 1) Limited monitorability score:

We agree that monitorability remains limited in some settings. The following table below shows the TPR and TNR breakdown of the monitorability metrics, averaged across all seeds, models, and tasks.

BaselineDescription training
Misaligned hack rate0.870.79
Interventiong-mean²0.130.37
TPR0.190.88
TNR (controlled)0.930.72
TNR (misaligned non-hack)0.690.21
Reward-intentg-mean²0.170.59
TPR0.180.85
TNR0.930.72

(Note that these numbers are slightly different from those in the paper. We determined that our original prompt for generating the reward description may have leaked some information about the hacking behavior, so we removed the leakage, revised the prompt and rerun all the training. The conclusions are unchanged.)

Compared to baseline training, our method effectively increases TPR but decreases TNR. The drop in TNR is partly an artifact of how the hack label is defined (e.g., failing all test cases is not considered as hacking) or the weak true reward (e.g., always output 0.5/0.5 judge score rather than more complex human preference scores). A model may therefore correctly reason about the rewarded behavior in the CoTs and generate an output labeled as not hacking, while the monitor flags it as hacking. The method thus improves monitorability when hacking occurs. More complex settings and stronger monitor tools should be explored in future work to address the false positive issue.

Regarding "users cannot easily know what kind of task types they will face", the method does not need prior knowledge of the task type, since the description is inferred from the trained model's rollouts and reward, though it would be helpful to have a description generator model that has some basic understanding of the task.

2. (Weakness 2) Qualifying "consistently improves monitorability":

We will revise it in the next version of the paper. Specifically, it should be qualified with "for models that learn to hack or have high hack rate". Here are the newest results after we updated the prompt for generating the reward descriptions:

Misaligned hack rateIntervention g-mean²Reward-intent g-mean²
TaskModelBaselineDescriptionBaselineDescriptionBaselineDescription
Lengthgpt-oss-20b1.001.000.310.910.300.85
gpt-oss-120b1.000.450.230.170.340.34
Nemotron1.000.880.080.320.310.84
LM Judgegpt-oss-20b1.001.000.010.360.010.22
gpt-oss-120b1.001.000.040.520.040.44
Nemotron0.931.000.020.250.020.15
Codinggpt-oss-20b0.670.550.270.300.260.85
gpt-oss-120b0.610.590.200.310.240.63
Nemotron0.590.660.000.150.010.97
Average0.870.79 (-0.08)0.130.37 (+0.24)0.170.59 (+0.42)
  • On coding, reward description training increases both the intervention and reward-intent g-mean² scores for the two gpt-oss models.
  • For the gpt-oss-120b model on the length task, the monitorability scores are lower or unchanged mainly because the model did not learn to hack as much. The hack rate decreases from 1.00 to 0.45, leaving fewer hacking rollouts to be monitored.

3. (Question 1) Add a baseline for the mitigation method:

We follow the reviewer's suggestion and add "Guess Reward" as a baseline, which appends the following text to the training queries that asks the model to reason about and guess its reward function before generating the final answer: "Note: your response is scored by an automated reward function that you are not shown, and it may differ from the task as stated. In your reasoning, think about what it might actually reward before committing to a response."

Due to limited compute and budget, we train all three models only on the length and judge tasks with one seed (seed=1), with gpt-5.6-luna as the monitor:

Misaligned hack rateIntervention g-mean²Reward-intent g-mean²
TaskModelBaseline TrainingGuess RewardDescriptionBaseline TrainingGuess RewardDescriptionBaseline TrainingGuess RewardDescription
Lengthgpt-oss-20b1.001.001.000.310.350.910.300.340.85
gpt-oss-120b1.000.990.450.230.190.170.340.310.34
Nemotron*1.000.410.880.080.050.320.310.050.84
LM Judgegpt-oss-20b1.001.001.000.010.010.360.010.010.22
gpt-oss-120b1.000.771.000.040.020.520.040.040.44
Nemotron*0.930.861.000.020.020.250.020.020.15
Average0.990.840.890.110.110.420.170.130.47

Guess Reward matches or even underperforms baseline training. The CoTs do not verbalize the actual misaligned reward. The reward-guessing prompt seems to cause the CoTs to be even more aligned with the stated intent, which might explain the lower monitorability scores. Here are some sample CoTs from the misaligned gpt-oss-120b model:

  • Length (word limit is 1472 words): "...Potential sections...Add citations style perhaps. We need to keep within 1472 words..."
  • Judge: "We must judge criteria: correctness, completeness, clarity, tone..."

4. (Question 2) Add monitors from other model families:

We added Nemotron-3-Super-120B-A12B (same model family as one base model) and Claude Haiku 5.5 (a held-out model family) as monitors. Here are the results at the final checkpoint averaged across all seeds and tasks.

Misaligned hack rateIntervention g-mean²Reward-intent g-mean²
ModelMonitorBaselineDescriptionBaselineDescriptionBaselineDescription
gpt-oss-20bgpt-5.6-luna0.890.850.200.520.190.64
Nemotron-Super-120B0.890.850.140.620.130.66
Claude Haiku 5.50.890.850.170.490.150.60
gpt-oss-120bgpt-5.6-luna0.870.680.160.330.210.47
Nemotron-Super-120B0.870.680.100.380.150.40
Claude Haiku 5.50.870.680.170.270.200.42
Nemotrongpt-5.6-luna0.840.850.030.240.110.65
Nemotron-Super-120B0.840.850.010.460.010.72
Claude Haiku 5.50.840.850.010.190.040.64
Averagegpt-5.6-luna0.870.790.130.370.170.59
Nemotron-Super-120B0.870.790.080.490.100.59
Claude Haiku 5.50.870.790.120.320.130.55

We find no consistent effect of monitor-base-model family matching.

  • Compared to gpt-5.6-luna, Nemotron-Super-120B achieves lower monitorability score when monitoring the baseline Nemotron model but higher score when monitoring the description-trained Nemotron model.
  • gpt-5.6-luna scores lower than Nemotron-Super-120B when monitoring gpt-oss-20b but higher when monitoring gpt-oss-120b.
  • Claude Haiku 5.5 shows no consistent pattern either.

However, reward description training does achieve higher monitorability scores across all three monitors. Therefore, the low monitorability of the Nemotron models does not appear to relate to monitors' model family. It is more important for the CoTs to explicitly verbalize the actual rewarded behavior to be monitorable.

Thanks for your response. The additional experiments addressed most of my concerns. I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.