- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper conducts a systematic empirical investigation into a widely‑accepted hypothesis in the field of CoT monitorability: as task difficulty increases and models become more reliant on their chain‑of‑thought, CoTs become easier for monitors to detect undesirable model behaviors.
Critical strengths
This paper directly challenges the hypothesis that CoT necessity equals better monitorability. Through empirical evidence, it reveals substantial limitations of this proposition, delivering important cautionary implications for the real‑world safety deployment of CoT monitoring.
Critical weaknesses
- One limitation is that all experiments in this paper are restricted to synthetic symbolic tasks. While such tasks enable fine‑grained control over task difficulty and CoT reliance, it remains unclear whether the conclusions generalize to open‑text and real‑world agent tasks. The behavioral patterns of model cheating and misalignment in real‑world scenarios may differ from those observed in synthetic tasks.
- All undesirable behaviors in this work are explicitly induced via prompts: grader‑hacking hints and adversarial instructions that demand hiding side‑task computations. Consequently, it remains an open question whether the three evasion strategies identified in the paper would emerge from genuine real‑world model misbehavior. In another words, without such adversarial prompts in real‑world deployments, would models even adopt these evasion tactics in the first place? This directly impacts the applicability of the conclusions drawn in this work.
Questions for authors
See Weaknesses.
2 comments
Reply to Author
Thank you for the rebuttal. My core concerns remain: results are limited to synthetic tasks and constructed prompt settings, without convincing evidence for spontaneous emergence in real-world agent deployments. I retain my rating of 2.
We thank the reviewer for their insightful review. We agree with the reviewer that our results deliver “important cautionary implications for the real‑world safety deployment of CoT monitoring” when depending on CoT necessity to achieve high monitorability.
Our results in Figure 6 and Table 2 shows that there exist strategies for monitor evasion that continue to work even when CoT reliance is high. The two strategies we observe that accomplish this are general enough that we believe would be applicable to models in real-world settings. Both are a type of prompt injection: one convinces the monitor to only consider a subset of the CoT, and the other convinces the monitor that the user prompt contained additional instructions that were not there.
We agree that additional strategies for monitor evasion surely exist, especially on open-text and real-world agent tasks which have different affordances that a model may use for monitor evasion. Our work is not claiming to taxonomize all monitor evasion strategies that models may use, but instead to show that there are some that continue to work even when CoT reliance is high.
We note that we do not prompt the model to use the grader hacking hint in our hint-based evaluations. We simply make the hint available to be used, and the model decides whether to use it or solve the task without it. As we state in Section 3.2, prior work has demonstrated that models are prone to reward hack by exploiting the grader. And we find that the models we study do use the hint a significant portion of the time (Figures 14 and 15).
We use the simple setting of leaking the hint as part of the prompt to isolate how monitorability varies with CoT reliance on the original task and in using the hint itself. In a real-world setting, a model might need to take additional steps before accessing the grader, such as searching for its implementation on the host machine or on an online repository. It would be difficult to create an evaluation where the difficulty in accessing the grader can be smoothly varied, though that would be a great experiment in observing the relationship between CoT reliance and monitorability. We suspect that such an experiment would show similar findings: when CoT reliance is low, the model can evade the monitor through non-verbalization. When CoT reliance is high, the model can evade the monitor through other strategies (Table 4).
Studying model misbehavior is difficult when the misbehavior is rare, making it hard to get enough instances to study. It is still valuable to study such misbehavior when it is sufficiently harmful, or if future models may exhibit the misbehavior at higher rates. There are two strategies generally available to induce higher rates of misbehavior for study: fine-tuning and prompting. For the secret-side-task evaluations, we use prompting because it effectively induces the target misbehavior (attempting the side task) and is less computationally expensive than fine-tuning. And as stated in the previous question, we think that the observed strategies for monitor evasion are not specific to the tasks we study, and can be applied in real-world tasks. And even though additional strategies are likely to be used for monitor evasion in other settings, our results show that we should not rely on high CoT reliance alone to deliver high monitorability.