- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates why reasoning models fail to follow formatting and lexical constraints on their thinking traces (e.g., using only uppercase letters or avoiding banned keywords) on the CoT-Control benchmark. The authors argue that a binary pass/fail score across an entire trace obscures whether models fail at the very beginning, revert to normal text midway, or fail simply due to generating longer traces.
First, the authors modify the benchmark graders to locate the exact word or sentence of the first violation in each thinking trace. Using this violation position, they apply survival analysis and Shapley attribution to separately measure compliance over the first 20 words, subsequent violation rates per 1,000 words, and the effect of trace length. Second, they prefill the beginning of the thinking block with 20 or 100 words from the model's unconstrained trace rewritten into the target style. They show that models with 0% benchmark scores can sustain the style when given a compliant opening, and extend this analysis across model sizes, post-training checkpoints, and thinking versus final-output channels
Critical strengths
-
Practical relevance. The problem of understanding why reasoning models fail to follow constraints on their thinking traces—and distinguishing immediate failure at the first word from later reversion—is practical and relevant to chain-of-thought evaluation.
-
Extensive empirical evaluation. The paper includes extensive experiments across multiple model families, model scales, and post-training checkpoints, along with useful controls such as evaluating prefilled prefixes without the constraint instruction.
Critical weaknesses
-
Extremely poor clarity and organization. The paper is exceptionally difficult to follow due to overly compressed prose, constant cross-references to appendix tables, and inconsistent definitions of "onset" across sections (switching between 20-word survival, first-word compliance, and fitted mixture parameters).
-
Unclear central contribution. The paper combines survival analysis, gamma-frailty fitting, Shapley attribution, prefix prefilling, prompt hinting, checkpoint comparisons, and reasoning-effort ablations without establishing a coherent central narrative.
-
Limited novelty and conceptual contribution. The proposed framework primarily applies standard statistical tools to an existing benchmark without introducing substantial methodological novelty or creating new benchmarks. Moreover, the finding that models continue generating in uppercase after an uppercase prefix (even without instructions) largely reflects basic in-context style continuation rather than a new insight into reasoning controllability.
-
Interpretation of the prefilling intervention. Prefilling 20 or 100 words does not isolate "onset alone": it alters the conditioning context, selects different surviving subpopulations, and still significantly changes subsequent violation rates in several settings.
Questions for authors
Please refer to the weaknesses
Thank you for the review. We take the clarity criticism seriously, since the other two reviewers raised it too.
Clarity and definitions. You are right that "onset" appears under three definitions. The revision will use one (first native unit complies) throughout, call the 20-word quantity "early survival", and keep the mixture fit only as an appendix check. We will also cut the appendix cross-references in the main text and bring the few numbers each claim relies on into it.
Central contribution. In one sentence: a 0% controllability score can mean two different things, "the model cannot keep the constraint" or "the model never starts", and these have different safety implications. We show how to tell them apart with a cheap diagnostic. The revision will be organised around that, with the other analyses in the appendix as support.
Novelty and style continuation. We agree that continuing an uppercase context is not surprising in itself, and the paper should say so more plainly. The no-instruction control exists exactly to measure it. What we think is new is what the control reveals: the share due to continuation varies widely (almost none in Qwen3-8B, most of the effect in Qwen3-32B), the OLMo-3 checkpoints with long-CoT SFT have lost it, and none of this is visible in a benchmark score that is 0% for all of them. On tools: the statistics are standard on purpose. The contribution is the measurement question, plus two concrete problems in the released benchmark (the "meow" loops scored as compliant, and the case-sensitive tag parser).
Prefilling does not isolate onset. Agreed, and we will stop claiming it does. The prefix changes the context and changes who survives. We will describe it as a probe that fixes the start so that what follows can be measured, report the cells where maintenance moves, and note that selection among adopters works against the contrasts we report, not in favour of them.