- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper evaluates the following: Through RL post-training, chain-of-thought length of reasoning models is shortened while maintaining performance on benchmarks. Then, the paper checks whether the CoTs verbalize hints or biases that are included in the prompt and influence the final prediction. The central result is that models with compressed CoTs verbalize these hints less often than the original model, although they change predictions, and this behavior is induced through the compression.
Critical strengths
The paper is very clearly written, the technical claims are well supported by the experiments, and the paper contains important ablations and checks, for example human validation of the LLM-judged hint verbalization, and the ablation that randomly deletes sentences. Hyperparameters, metrics, and design decisions are well motivated. From the technical point of view, this is a good paper.
Critical weaknesses
I see two important weaknesses that support rejection. One is about the "monitorability" construct, the other about the conclusion that compressed CoTs are less faithful.
(W1) The central problem of the paper is to assume that an instrument (hint verbalization), which was used in prior work to make an existence claim, is a general method to "monitor" CoT faithfulness. Prior work (Turpin et al., 2023 and Chen at al., 2025), i.e., the main sources for the hint interventions, make the following claim:
"Hints are the only change to the prompt" and "Hints change the prediction" and "Hints are not verbalized" The CoT does not contain surface all (internal) reasoning that determines the prediction
However, none of these works propose using hint verbalization to interpret arbitrary CoTs, for example, if we want to understand post-hoc whether an LLM prediction was determined only by reasons stated in the CoT. Unfortunately, the paper sounds like it assumes this:
removing the evidence monitors depend on (line 26)
Monitoring it catches sabotage that action-only monitors miss (line 107)
In this light, the paper appears to allow two readings:
- Compressed CoTs are less accessible to in-deployment monitoring of misleading reasoning.
This is not supported by the experiments in the paper, because it is only about hint verbalization, i.e., a counterfactual prompt intervention that can only provide evidence for model properties, not specific traces. - If all (tested) LLMs are trained for CoT compression, we will observe even lower verbalization rates than reported by Turpin et al. or Chen et al.
This is supported by the experiments, but it mainly affirms the results in previous work, while adding limited additional insight, because it has already been established that CoTs omit important information.
If we accept reading (1), the paper's main claims are not supported by the experiments. If we accept reading (2), the paper's significance is questionable. In either case, I find this a strong argument against acceptance.
(W2) A second problem is that the implicit claim of Sec. 6 is that CoT compression teaches LLMs to output less faithful traces. There is some evidence in favor, such as the ablation from dropping random sentences and RL training without length penalty, but it is not clear to what extent this is really related to lower faithfulness of the traces. For example, the model may have learned (possibly benign) behaviors such as repeating less information from the prompt in general, which then by accident also includes the hints used in the setup in the paper. This requires either more transparency about the limitations, or additional experiments.
Questions for authors
(Q1) Is there any evidence that the hint verbalization monitor supports any broader CoT interpretation or monitorability that could be used in practice, i.e., in deployment?
(Q2) Can we establish a more direct link between CoT faithfulness (in general, beyond hint verbalization) and CoT compression training?