acceptodds
Under review as a conference paper at ICLR 2027

Length Penalties Make Chain-of-Thought Less Monitorable

Abstract

Recent work trains reasoning models with length penalties to curb overthinking and cut inference cost. We show that these penalties make the chain of thought less monitorable. A length-compressed model still lets misleading hints steer its answers, but it less often verbalizes their influence. We train Qwen3-4B and Qwen3-14B with reinforcement learning under length penalties targeting 60% down to 30% of baseline chain-of-thought length, then evaluate them with nine types of biasing hints on held-out MMLU-Pro-R and four transfer benchmarks. A chain is faithful when an LLM monitor can tell from it that the hint influenced the answer. At the 30% target, accuracy stays near baseline and wrong-answer hints switch answers as often as before. Yet faithfulness drops on every evaluation set for both models, by 39% for Qwen3-14B and 35% for Qwen3-4B on MMLU-Pro-R. A control trained with the same correctness and format rewards but no length penalty leaves faithfulness intact or raises it. Shortening alone does not explain the drop. Compressed chains mention the hint 7 to 35 percentage points less often than the uncompressed model’s chains shortened to the same length by random sentence deletion, across both model sizes and all five evaluation sets. Length penalties therefore trade monitorability for inference cost by removing the evidence monitors depend on.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
chain-of-thought, reasoning models, length penalties, monitorability, faithfulness, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x7F6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper evaluates the following: Through RL post-training, chain-of-thought length of reasoning models is shortened while maintaining performance on benchmarks. Then, the paper checks whether the CoTs verbalize hints or biases that are included in the prompt and influence the final prediction. The central result is that models with compressed CoTs verbalize these hints less often than the original model, although they change predictions, and this behavior is induced through the compression.

Critical strengths

The paper is very clearly written, the technical claims are well supported by the experiments, and the paper contains important ablations and checks, for example human validation of the LLM-judged hint verbalization, and the ablation that randomly deletes sentences. Hyperparameters, metrics, and design decisions are well motivated. From the technical point of view, this is a good paper.

Critical weaknesses

I see two important weaknesses that support rejection. One is about the "monitorability" construct, the other about the conclusion that compressed CoTs are less faithful.


(W1) The central problem of the paper is to assume that an instrument (hint verbalization), which was used in prior work to make an existence claim, is a general method to "monitor" CoT faithfulness. Prior work (Turpin et al., 2023 and Chen at al., 2025), i.e., the main sources for the hint interventions, make the following claim:

"Hints are the only change to the prompt" and "Hints change the prediction" and "Hints are not verbalized" →\rightarrow The CoT does not contain surface all (internal) reasoning that determines the prediction

However, none of these works propose using hint verbalization to interpret arbitrary CoTs, for example, if we want to understand post-hoc whether an LLM prediction was determined only by reasons stated in the CoT. Unfortunately, the paper sounds like it assumes this:

removing the evidence monitors depend on (line 26)
Monitoring it catches sabotage that action-only monitors miss (line 107)

In this light, the paper appears to allow two readings:

  1. Compressed CoTs are less accessible to in-deployment monitoring of misleading reasoning.
    This is not supported by the experiments in the paper, because it is only about hint verbalization, i.e., a counterfactual prompt intervention that can only provide evidence for model properties, not specific traces.
  2. If all (tested) LLMs are trained for CoT compression, we will observe even lower verbalization rates than reported by Turpin et al. or Chen et al.
    This is supported by the experiments, but it mainly affirms the results in previous work, while adding limited additional insight, because it has already been established that CoTs omit important information.

If we accept reading (1), the paper's main claims are not supported by the experiments. If we accept reading (2), the paper's significance is questionable. In either case, I find this a strong argument against acceptance.


(W2) A second problem is that the implicit claim of Sec. 6 is that CoT compression teaches LLMs to output less faithful traces. There is some evidence in favor, such as the ablation from dropping random sentences and RL training without length penalty, but it is not clear to what extent this is really related to lower faithfulness of the traces. For example, the model may have learned (possibly benign) behaviors such as repeating less information from the prompt in general, which then by accident also includes the hints used in the setup in the paper. This requires either more transparency about the limitations, or additional experiments.

Questions for authors

(Q1) Is there any evidence that the hint verbalization monitor supports any broader CoT interpretation or monitorability that could be used in practice, i.e., in deployment?

(Q2) Can we establish a more direct link between CoT faithfulness (in general, beyond hint verbalization) and CoT compression training?

Reviewer_1jjsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates the tradeoff between chain of thought compression (via a length penalty) and CoT faithfulness (monitorability). Using hint interventions on models that have been trained with varying degrees of length compression, they measure faithfulness of the resulting chains as the proportion of chains where the model is influenced by the hint and reports that influence. They find that length compression reduces this faithfulness metric across multiple model scales, and that this transfers to other benchmarks (MathQA, etc.). They also find that this is not an effect of generic RL training, and that length compression also reduces mentions of reasoning behaviors (e.g. verification, backtracking, exploration). Overall, this work identifies a potential safety vulnerability induced by a common RL post-training technique (length penalties) and contributes towards efforts to maintain CoT monitorability as we train models for other desirable behaviors.

Critical strengths
  1. The paper identifies an interesting problem – a tradeoff between length compression and monitorability in CoTs – that is both consequential for safety research and has not been addressed fully by prior research. The main finding is broadly relevant for both safety researchers and RL practitioners.

  2. The paper’s methodologies are generally sound; they include appropriate controls and sanity checks (non-length-penalized RL baseline, automatic monitor checks), and their metrics are carefully designed. The results are therefore straightforward to understand and appear reliable.

Critical weaknesses
  1. While the main result is interesting and the experiments are generally well thought out, I find the paper to be lacking somewhat in insight. The authors do not identify a mechanism for the decrease in faithfulness, nor do they introduce a method to correct it. The work would be greatly strengthened if the authors either provided deeper insight into why the drop in faithfulness occurs (at what point in RL training does it occur? Is the model still aware internally that it has been influenced by a hint? Is there something about the reward structure that causes models to drop hint mentions in particular rather than other CoT content? What kinds of content do compressed chains preserve?) and/or a method for maintaining faithfulness under length compression. As it stands, it’s hard to know what to take away from the paper.

  2. I was somewhat confused by Section 6. I wasn’t sure how random sentence deletion establishes that brevity is not the cause of decreasing hint mentions, since it’s quite plausible that models learn not to mention hints for the sake of brevity (compression). The two alternative explanations (“Does a compressed chain drop hint mentions because it is short, or because of what training taught it to keep?”) could use more explanation, since I don’t actually know how these differ (doesn’t the model have to discard information because the chain is short?). Maybe a more fitting control would have been to train models with length compression and a reward that preserves hint mentions, since this would establish that short CoTs can still have room for hint mentions.

Questions for authors

Clarification requests:

  1. Can the authors provide more justification for the random sentence deletion experiment in Section 6?
  2. Did the authors perform any evaluations across training checkpoints or more detailed CoT analysis (i.e., what kinds of information do compressed CoTs preserve? What tends to take up space in the compressed CoTs?)

Suggestions for future work:

  1. Can the authors provide insight into the mechanism behind the drop in faithfulness, or ways to potentially fix the problem?
Reviewer_pP8COpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how length-penalized RL affects CoT monitorability. It finds that compressed traces disclose hint influence less often, while task accuracy and hint influence remain largely unchanged.

Critical strengths

The no-penalty RL control and matched-switch analysis help isolate the effect of length penalties, making the evidence more convincing than a simple correlation between trace length and monitorability.

Critical weaknesses
  1. The novelty claim is too strong. Arora & Zanette[1] already observed reduced faithfulness after compression. The additional controls and evaluations are useful, but “the first positive result” does not fairly reflect this prior finding.
  2. The random-deletion baseline does not fully support the interpretation. Learned compression may omit explanations that are unnecessary for answering the question. It is unclear whether hint disclosures are removed disproportionately compared with other types of content.

[1] Arora D, Zanette A. Training language models to reason efficiently[J]. Advances in Neural Information Processing Systems, 2026, 38: 60770-60808.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.