- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how reasoning efficiency training affects the faithfulness and monitorability of chain-of-thought (CoT). It evaluates three efficiency methods across three base models, measuring faithfulness through Normalized Simulatability Gain (NSG) and monitorability through the detection of sycophancy and cognitive-bias interventions from CoT alone. NSG decreases in 11 of 15 training settings, which the authors attribute primarily to reduced behavioral consistency rather than loss of explanatory information. Monitorability is comparatively robust: sycophancy remains detectable despite substantial CoT shortening, while cognitive-bias detection deteriorates under the strongest compression. Some settings substantially shorten CoT without reducing NSG, suggesting that the effects of efficiency training depend on the model, training method, task, and intended use of CoT.
Critical strengths
-
Distinguishes the predictive value of CoT from its monitorability. Evaluating these separately shows that reasoning efficiency training can affect different oversight functions differently. Comparisons across multiple base models, training methods, and tasks provide useful evidence that shortening CoT does not uniformly undermine oversight.
-
Investigates behavioral consistency and concrete failure patterns beyond CoT length. The paper identifies settings that substantially shorten CoT without reducing NSG, examines the association between consistency and NSG, and illustrates cue omission and reframing. These analyses provide useful hypotheses about when efficiency training compromises oversight.
Critical weaknesses
-
The attribution of NSG degradation primarily to inconsistency is insufficiently established. The ceiling estimated from ten samples per question is vulnerable to finite-sample bias, while the decomposition assumes that trained models retain the base model’s fraction of the attainable NSG ceiling. These analyses do not adequately separate stochasticity, loss of explanatory information, and predictor limitations.
-
The conclusions exceed what the evaluations directly establish. Excluding stronger predictors limits the robustness of the NSG findings, while evaluation with one monitor on restricted tasks provides limited support for broader safety claims. Lower NSG does not directly establish worse identification of decision-relevant features, and preserved monitorability scores do not guarantee accurate detection of causal influence in individual responses.
Questions for authors
-
Do the changes in NSG before and after efficiency training, as well as the rankings of efficiency methods, persist when stronger predictors are included? Given that NSG normalizes gains by the remaining baseline error, why were higher no-CoT accuracy and smaller uplift considered sufficient reasons to exclude stronger predictors? Addressing this would clarify whether the findings generalize across predictor capabilities or depend on the selected predictors’ ability to interpret compressed CoT.
-
Are independent samples used to select the oracle’s modal answer and evaluate its accuracy? Do the estimated ceilings and differences between base and efficiency-trained models stabilize as the number of samples increases beyond ten? Addressing this would establish how robust the ceiling estimates and their changes are to finite-sample bias and sampling variability.
-
Can changes in the NSG ceiling be decomposed into the contributions of changes in reference-model consistency and changes in no-CoT predictor accuracy? This decomposition would distinguish a lower ceiling caused by increased inconsistency from a change in the baseline used to normalize NSG.
-
Does the conclusion that inconsistency explains most of the NSG degradation remain valid when the assumption that trained models retain the base model’s fraction of the attainable ceiling is relaxed? Addressing this would show whether the main attribution is robust or depends strongly on the decomposition assumption.
-
Are answer-changing cases selected using individual outputs, or are they identified through repeated sampling that establishes stable changes in answer distributions? Do the findings hold when the same question set is used before and after training? Addressing this would help distinguish systematic behavioral changes from sampling variability and changes in the composition of the analyzed cases.
-
Can the authors report the intervention effect, the unadjusted detection rate, the true negative rates in both conditions, and the number of eligible questions before and after training? Do the monitorability results hold on the same question set? Addressing this would clarify whether stable aggregate scores reflect preserved detection ability or changes in intervention strength, metric components, or question eligibility.
5 comments
2: "The conclusions exceed what the evaluations directly establish…"
We address all of the individual points regarding this weakness below.
"Excluding stronger predictors limits the robustness of the NSG findings" and "Given that NSG normalizes gains by the remaining baseline error, why were higher no-CoT accuracy and smaller uplift considered sufficient reasons to exclude stronger predictors?"
We agree that excluding them wasn't necessary, however, including them does not change our findings. To compare our original results to a stronger model, we add DeepseekV4-Flash as a predictor for the base and GLP results on Qwen3-4B. Table 2 shows that the reduction in NSG with DeepseekV4-Flash is similar to that of the predictors we originally used, both separately and pooled. Given this, we argue that our original results are robust to swapping in different predictors, including stronger models like DeepseekV4-Flash.
| Predictor | ΔNSG |
|---|---|
| Pooled (paper) | -9.5 [-12.3, -6.8] |
| Qwen3-32B | -12.5 [-16.1, -8.8] |
| GPT-OSS-20B | -8.2 [-12.0, -4.5] |
| Gemma3-27B | -8.3 [-12.2, -4.1] |
| DeepseekV4-Flash | -11.0 [-15.2, -6.8] |
Table 2. Change in NSG after GLP training on Qwen3-4B for each predictor, with 95% confidence intervals.
"Evaluation with one monitor"
To address the fact that we only use one monitor, we re-run the monitorability evaluation using GPT-6-Luna (replacing DeepseekV4-Flash) for both FACE-Eval and the medical cognitive-bias task. We observe the same two patterns. Sycophancy detection on FACE-Eval is preserved under compression (max drop -0.08, matching the main text), and cognitive-bias detection degrades with compression, correlating with CoT length relative to the base model. Table 3 shows a sample of the results for the cognitive-bias task, and the full results are in Appendix D of the revised paper.
| Model | Method | DeepseekV4-Flash | GPT-6-Luna |
|---|---|---|---|
| Qwen3-8B | Base | 0.868 | 0.925 |
| Qwen3-8B | L1-Low | 0.547 | 0.589 |
| Olmo3-7B | Base | 0.773 | 0.850 |
| Olmo3-7B | L1-Low | 0.531 | 0.579 |
| Qwen3-4B | Base | 0.889 | 0.984 |
| Qwen3-4B | L1-Low | 0.617 | 0.634 |
Table 3. Cognitive-bias monitorability with the original monitor (DeepseekV4-Flash) vs GPT-6-Luna. Absolute scores change, but the same patterns can be observed when comparing base to trained models.
"Preserved monitorability scores do not guarantee accurate detection of causal influence in individual responses."
We argue that this is true of most metrics. For example, higher accuracy does not guarantee that a model is correct on a particular problem, but when we find higher accuracy we accept that the model is better on that task. In the same way, higher monitorability scores do not guarantee detection in individual responses, but they do show that in general the CoT is more monitorable than one with a lower score.
"Can the authors report the intervention effect, the unadjusted detection rate, the true negative rates in both conditions, and the number of eligible questions before and after training?"
Intervention effects can already be found in Appendix C.3. We have now also included the true positive rate, the true negative rates with and without the cue, and the number of eligible questions in the revised paper (Appendix C.4). The true negative rate without the cue stays close to 1 throughout, and on cognitive bias both true negative rates stay above 0.89, so where monitorability falls, it is because the true positive rate falls. The number of eligible questions before and after training is stable on sycophancy and increases on the cognitive-bias task (for example, from 39 to 58 of 150 for Qwen3-4B L1-Low on the false-consensus cue). These extra metrics show that the stable aggregate scores reflect preserved detection, and are not caused by changes in intervention strength, metric components or question eligibility.
"Do the monitorability results hold on the same question set?"
Yes, the results hold. To check this, we took the intersection of eligible questions for each base and trained model pair, and recomputed monitorability for both models on these shared questions. The change in monitorability from the base model on the shared questions is close to the change on all questions in every setting, and every drop we initially reported remains (Appendix C.4, Table 6 in the revised paper). This is expected, as the overlap between the eligible questions of the base and trained models is also very high (Appendix C.4, Table 5).
Thank you for the additional analyses. The independent oracle evaluation, the 30-sample check, and the sensitivity analyses in Appendix E substantially address my concerns about the NSG results and the role of answer consistency.
My remaining concern is the scope of the monitorability conclusions. The second monitor and shared-question analysis strengthen the benchmark findings, but the reported TPR is adjusted for cue strength. Could you consider reporting the unadjusted flag rate on the shared eligible questions and framing the conclusion in §7 around the monitorability scores observed on the evaluated tasks? If you think the broader “mostly safe” wording is warranted, I would appreciate clarification on what evidence supports it.
Thank you for your detailed review and constructive feedback. Please find the answers to your concerns below.
1: "The attribution of NSG degradation primarily to inconsistency is insufficiently established…"
We address the points regarding this individually:
"The ceiling estimated from ten samples per question is vulnerable to finite-sample bias"
To test this, we resampled the base and GLP models 30 times per question instead of 10, on a fixed random fifth of the counterfactuals that is the same for every model. Using 30 samples, the oracle upper bound still falls for Qwen3-4B and Olmo3-7B and has a very small positive change for Qwen3-8B (Table 1). Given these results, we are confident that increasing the number of samples will not change our claims about the change in the upper bound.
Table 1. Change in the NSG upper bound (pts) from base to GLP, estimated from 10 vs 30 samples per question.
"Are independent samples used to select the oracle's modal answer and evaluate its accuracy?"
Yes they are independent samples. The oracle's most common answer comes from the 10 resamples, and its accuracy is scored against the model's original answer. That original answer was generated separately from the resamples, and it is the same answer the predictors are scored against.
"Does the conclusion that inconsistency explains most of the NSG degradation remain valid when the assumption that trained models retain the base model's fraction of the attainable ceiling is relaxed?"
To relax this assumption, we also recalculate the share using the trained model's fraction of the bound instead of the base model's. This asks how much NSG the trained model would recover if its upper bound were restored to the base model's level while it kept its own fraction of the bound. Using this, we find that the share of the NSG reduction explained by the fall in the upper bound changes only slightly, from 24-80% to 20-74%, and remains at or above 50% in 8 of the 11 runs with an NSG drop, the same 8 as before (Appendix E, Table 8 in the revised paper). The main exception under both assumptions is Qwen3-4B L1-Low and L1-Medium, where most of the reduction is not explained by the bound.
"predictor limitations"
To show that a stronger predictor would not change the results, we include DeepseekV4-Flash as a predictor. We discuss this further in our response to the second weakness (see point 2 below).
"Can changes in the NSG ceiling be decomposed into the contributions of changes in reference-model consistency and changes in no-CoT predictor accuracy?"
Yes, we can decompose this. We would also like to highlight that when no-CoT accuracy falls, it raises the upper bound rather than lowering it, so it cannot explain the fall in the bound. To decompose this, we split each change in the bound into a part from consistency and a part from no-CoT accuracy (Appendix E, Table 9). In every setting with a significant NSG drop, the no-CoT part either raises the bound or barely changes it (Olmo3-7B, -0.9 to -0.3 points). The fall in the upper bound therefore comes from the fall in consistency.