- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper examines what two reasoning diagnostics actually measure in Qwen3's same-weight thinking and non-thinking modes. Instance A analyzes the leading variance share of the cross-problem hidden-state difference matrix and finds that its dominant direction is strongly aligned with the between-problem mean, suggesting that the resulting curve mainly reflects changes in shared versus question-specific energy rather than reasoning-specific rank structure. The authors further investigate the crossing position using controlled continuations, including a zero-content prefix that reproduces a similar crossing position without reasoning content.
Instance B examines binary agreement between verbalized confidence and a linear probe, showing algebraically that the agreement indicator reduces to the probe's hard label whenever verbalized confidence exceeds the threshold. Experiments across six Qwen3 model sizes on MATH500 support these observations, while several preregistered hypotheses, including the difficulty-axis and steering analyses, are not supported.
Critical strengths
-
The paper provides a detailed empirical analysis of the geometry underlying Instance A. In particular, the strong alignment between the leading direction and the between-problem mean is supported by both in-sample and question-disjoint split-half measurements (Table 1, L163–201). The authors also distinguish mathematical identities from empirical findings in Instance B (L350–367) and transparently document negative results and corrections. These efforts are valuable because they encourage more careful interpretation of representation-level diagnostics and reduce the risk of attributing computational meaning to potentially misleading measurements.
-
The content-graded ladder and zero-content prefix experiments provide meaningful evidence that the crossing position can be strongly influenced by formatting and token alignment rather than reasoning onset (Table 2, L209–269; L288–294). In particular, the finding that a zero-content prefix produces a crossing position close to the original thinking/non-thinking contrast is a useful counterexample to interpreting this position as the beginning of reasoning. I consider this one of the more convincing empirical contributions of the paper.
Critical weaknesses
- The motivation and novelty of the two diagnostic audits are not sufficiently established.
My primary concern is that the paper does not convincingly establish that the two specific statistics being audited are widely used reasoning diagnostics or that the interpretations being challenged are established practices in the literature. The title and abstract describe them as "standard reasoning diagnostics" (L002–016), while the introduction claims that they measure something other than what they are supposed to measure (L041–053). However, I could not identify clear references demonstrating that: The particular statistic has been established as a measure of reasoning-related rank structure or a computational phase transition. Binary agreement between thresholded verbal confidence and a probe has been established as a standard measure of divergence between confidence estimators. The related work mainly discusses adjacent findings about format confounding, reasoning-mode probes, and thinking/non-thinking representations (L060–079), rather than establishing the specific measurement practices being criticized. The Instance B discussion also lacks engagement with the broader literature on verbalized confidence and calibration.
This distinction matters for the paper's contribution. Auditing a widely adopted but systematically misinterpreted diagnostic would be an important methodological contribution. However, identifying limitations in two particular statistical constructions, without establishing their relevance to existing research practice, is a substantially narrower result. Moreover, several conclusions overlap with prior observations already discussed by the authors, particularly the importance of format directions (Sahoo et al., 2026) and the template-level effects of the thinking switch (Yang et al., 2026; L060–070). The paper offers additional quantitative evidence, but the conceptual advance beyond these findings remains unclear.
- The main technical findings are relatively elementary, while some interpretations exceed the available evidence.
For Instance A, the leading variance share is computed from an uncentered Gram matrix (L093–097), while the shared energy fraction measures the contribution of the between-problem mean (L182–191). By the Rayleigh quotient inequality, the shared energy fraction cannot exceed the leading variance share. Therefore, the observation that this inequality holds in all 120 model-position combinations (L197–199) is a mathematical consequence of the definitions, rather than an independent empirical finding. Similarly, when the mean component already accounts for 95–99% of the energy, strong alignment between the leading direction and the mean is largely expected.
The paper would benefit substantially from directly comparing the original curve with the leading variance share and effective rank computed after centering the difference matrix. Although Figure 2 reports effective rank, it does not adequately establish what nontrivial rank structure remains after removing the shared mean component. I also find the interpretation that a question-invariant direction "can only be template or format" (L189–190) too strong. A shared direction could also encode task-level processing, global reasoning-mode information, or other question-independent computational signals. The non-thinking controls demonstrate that the observed decay is not unique to reasoning, but they do not establish that the shared direction consists exclusively of formatting information. For Instance B, the central result follows directly from the Boolean identity:
$[ G=\mathbf{1}[\mathbf{1}[a>0.5]=\mathbf{1}[b>0.5]]. ]$
When (a>0.5), (G) equals the probe's hard label; otherwise, it equals its complement. Thus, the equality between the number of disagreements with the probe label and the number of observations with is guaranteed algebraically. The authors correctly acknowledge this, but presenting the identity and its numerical verification as a major technical contribution offers limited additional insight. The empirical value lies instead in demonstrating when confidence saturation actually occurs and how it affects the usefulness of the agreement statistic. However, the experimental support is limited: Instance B has usable results from only three model sizes, with relatively small effective samples (Table 4, L325–335). Furthermore, the main text does not sufficiently specify the verbal-confidence elicitation procedure, probe training protocol, sample filtering, or the prediction targets underlying the reported AUROCs. The discussion also appears to conflate two distinct observations: agreement may become redundant with the probe's prediction, and agreement may perform poorly as a predictor of answer correctness. The latter does not, by itself, demonstrate that agreement is an invalid measure of divergence.
These limitations are compounded by the narrow experimental setting (one model family, one dataset, and the first 24 continuation tokens at three selected layers; L134–156), which makes it difficult to assess how broadly the conclusions apply. Finally, the paper contains some inconsistencies between its abstract and final analysis. The abstract states that crossing positions show no monotone scaling across six model sizes (L025–026), whereas Section 5.3 explicitly concludes that neither the presence nor absence of scaling is established (L298–307). The appendix further identifies the abstract's formulation as retired wording (L648–650). Although the authors transparently disclose exploratory and post-hoc analyses, these should be clearly distinguished from preregistered confirmatory results in the final presentation. Overall, I appreciate the careful auditing effort and the informative controlled experiments. However, the strongest findings are either consequences of basic mathematical properties or relatively narrow empirical observations, while the broader interpretations and claimed methodological significance are not sufficiently supported.
Questions for authors
-
Could the authors identify specific prior papers and equations in which the two audited statistics are used and interpreted in the ways criticized here? In particular, what evidence establishes that these are standard reasoning diagnostics rather than statistics specific to a limited experimental protocol?
-
Could the authors report the leading variance share and effective rank after centering by subtracting the between-problem mean, alongside the original results? How much of the observed curve is attributable to the uncentered mean component, and what rank-related information remains after centering?
-
Could the authors clarify how verbal confidence (a) is elicited, which probe produces (b), how the probe is trained and evaluated, and how the effective sample sizes are obtained? Please also specify the prediction targets for AUC(hard) and AUC(agree), and distinguish the agreement statistic's redundancy from its performance in predicting answer correctness.
-
What evidence supports attributing the question-invariant mean direction specifically to formatting, rather than to other shared model states or task-level computations? Would the main conclusions still hold under the more conservative interpretation that the statistic cannot distinguish these possibilities?
-
Could the authors provide an accessible, timestamped preregistration and distinguish the originally specified confirmatory analyses from subsequent exploratory analyses? In particular, which scaling-related statement should be regarded as the final conclusion, given the inconsistency between the abstract and Section 5.3?