acceptodds
Under review as a conference paper at ICLR 2027

The Ruler Decides: Two Standard Reasoning Diagnostics Measure Other Than Claimed

Abstract

Same-weight dual-mode models such as Qwen3 offer a controlled pairing: one weight set, one prompt, and only an `enable_thinking` switch. We audit two standard reasoning diagnostics on this pairing: the leading variance share of the cross-problem difference matrix , and binary agreement read as metrics of divergence. In this paper, we ask what each actually measures. On MATH500 (0.6B–32B) the leading direction is mainly the question-invariant (mean) direction: **at L8** across the transition (; 0.6B–14B; question-disjoint split-half). The familiar curve is therefore a **question-specific energy-share curve**, not a measure of rank structure; a non-thinking control shows the same drop, so its existence is not evidence of a phase transition. We build a content-graded ladder of five controlled continuations. Its crossing position orders consistently with the contrast's real content, but monotonicity does not survive a family-wise joint test. That position shows no monotone scaling across six sizes, and a pre-registered difficulty axis fails outright.

Primary area:
interpretability and explainable AI
Keywords:
LLM reasoning, representation analysis, measurement validity, reasoning diagnostics, think/no-think contrast

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ai1WOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines what two reasoning diagnostics actually measure in Qwen3's same-weight thinking and non-thinking modes. Instance A analyzes the leading variance share of the cross-problem hidden-state difference matrix and finds that its dominant direction is strongly aligned with the between-problem mean, suggesting that the resulting curve mainly reflects changes in shared versus question-specific energy rather than reasoning-specific rank structure. The authors further investigate the crossing position using controlled continuations, including a zero-content prefix that reproduces a similar crossing position without reasoning content.

Instance B examines binary agreement between verbalized confidence and a linear probe, showing algebraically that the agreement indicator reduces to the probe's hard label whenever verbalized confidence exceeds the threshold. Experiments across six Qwen3 model sizes on MATH500 support these observations, while several preregistered hypotheses, including the difficulty-axis and steering analyses, are not supported.

Critical strengths
  • The paper provides a detailed empirical analysis of the geometry underlying Instance A. In particular, the strong alignment between the leading direction and the between-problem mean is supported by both in-sample and question-disjoint split-half measurements (Table 1, L163–201). The authors also distinguish mathematical identities from empirical findings in Instance B (L350–367) and transparently document negative results and corrections. These efforts are valuable because they encourage more careful interpretation of representation-level diagnostics and reduce the risk of attributing computational meaning to potentially misleading measurements.

  • The content-graded ladder and zero-content prefix experiments provide meaningful evidence that the crossing position can be strongly influenced by formatting and token alignment rather than reasoning onset (Table 2, L209–269; L288–294). In particular, the finding that a zero-content prefix produces a crossing position close to the original thinking/non-thinking contrast is a useful counterexample to interpreting this position as the beginning of reasoning. I consider this one of the more convincing empirical contributions of the paper.

Critical weaknesses
  1. The motivation and novelty of the two diagnostic audits are not sufficiently established.

My primary concern is that the paper does not convincingly establish that the two specific statistics being audited are widely used reasoning diagnostics or that the interpretations being challenged are established practices in the literature. The title and abstract describe them as "standard reasoning diagnostics" (L002–016), while the introduction claims that they measure something other than what they are supposed to measure (L041–053). However, I could not identify clear references demonstrating that: The particular statistic (1−top1var(ωj))(1-\mathrm{top1_var}(\omega_j)) has been established as a measure of reasoning-related rank structure or a computational phase transition. Binary agreement between thresholded verbal confidence and a probe has been established as a standard measure of divergence between confidence estimators. The related work mainly discusses adjacent findings about format confounding, reasoning-mode probes, and thinking/non-thinking representations (L060–079), rather than establishing the specific measurement practices being criticized. The Instance B discussion also lacks engagement with the broader literature on verbalized confidence and calibration.

This distinction matters for the paper's contribution. Auditing a widely adopted but systematically misinterpreted diagnostic would be an important methodological contribution. However, identifying limitations in two particular statistical constructions, without establishing their relevance to existing research practice, is a substantially narrower result. Moreover, several conclusions overlap with prior observations already discussed by the authors, particularly the importance of format directions (Sahoo et al., 2026) and the template-level effects of the thinking switch (Yang et al., 2026; L060–070). The paper offers additional quantitative evidence, but the conceptual advance beyond these findings remains unclear.

  1. The main technical findings are relatively elementary, while some interpretations exceed the available evidence.

For Instance A, the leading variance share is computed from an uncentered Gram matrix (L093–097), while the shared energy fraction measures the contribution of the between-problem mean (L182–191). By the Rayleigh quotient inequality, the shared energy fraction cannot exceed the leading variance share. Therefore, the observation that this inequality holds in all 120 model-position combinations (L197–199) is a mathematical consequence of the definitions, rather than an independent empirical finding. Similarly, when the mean component already accounts for 95–99% of the energy, strong alignment between the leading direction and the mean is largely expected.

The paper would benefit substantially from directly comparing the original curve with the leading variance share and effective rank computed after centering the difference matrix. Although Figure 2 reports effective rank, it does not adequately establish what nontrivial rank structure remains after removing the shared mean component. I also find the interpretation that a question-invariant direction "can only be template or format" (L189–190) too strong. A shared direction could also encode task-level processing, global reasoning-mode information, or other question-independent computational signals. The non-thinking controls demonstrate that the observed decay is not unique to reasoning, but they do not establish that the shared direction consists exclusively of formatting information. For Instance B, the central result follows directly from the Boolean identity:

$[ G=\mathbf{1}[\mathbf{1}[a>0.5]=\mathbf{1}[b>0.5]]. ]$

When (a>0.5), (G) equals the probe's hard label; otherwise, it equals its complement. Thus, the equality between the number of disagreements with the probe label and the number of observations with (a≤0.5)(a\leq0.5) is guaranteed algebraically. The authors correctly acknowledge this, but presenting the identity and its numerical verification as a major technical contribution offers limited additional insight. The empirical value lies instead in demonstrating when confidence saturation actually occurs and how it affects the usefulness of the agreement statistic. However, the experimental support is limited: Instance B has usable results from only three model sizes, with relatively small effective samples (Table 4, L325–335). Furthermore, the main text does not sufficiently specify the verbal-confidence elicitation procedure, probe training protocol, sample filtering, or the prediction targets underlying the reported AUROCs. The discussion also appears to conflate two distinct observations: agreement may become redundant with the probe's prediction, and agreement may perform poorly as a predictor of answer correctness. The latter does not, by itself, demonstrate that agreement is an invalid measure of divergence.

These limitations are compounded by the narrow experimental setting (one model family, one dataset, and the first 24 continuation tokens at three selected layers; L134–156), which makes it difficult to assess how broadly the conclusions apply. Finally, the paper contains some inconsistencies between its abstract and final analysis. The abstract states that crossing positions show no monotone scaling across six model sizes (L025–026), whereas Section 5.3 explicitly concludes that neither the presence nor absence of scaling is established (L298–307). The appendix further identifies the abstract's formulation as retired wording (L648–650). Although the authors transparently disclose exploratory and post-hoc analyses, these should be clearly distinguished from preregistered confirmatory results in the final presentation. Overall, I appreciate the careful auditing effort and the informative controlled experiments. However, the strongest findings are either consequences of basic mathematical properties or relatively narrow empirical observations, while the broader interpretations and claimed methodological significance are not sufficiently supported.

Questions for authors
  • Could the authors identify specific prior papers and equations in which the two audited statistics are used and interpreted in the ways criticized here? In particular, what evidence establishes that these are standard reasoning diagnostics rather than statistics specific to a limited experimental protocol?

  • Could the authors report the leading variance share and effective rank after centering (ωj)(\omega_j) by subtracting the between-problem mean, alongside the original results? How much of the observed curve is attributable to the uncentered mean component, and what rank-related information remains after centering?

  • Could the authors clarify how verbal confidence (a) is elicited, which probe produces (b), how the probe is trained and evaluated, and how the effective sample sizes are obtained? Please also specify the prediction targets for AUC(hard) and AUC(agree), and distinguish the agreement statistic's redundancy from its performance in predicting answer correctness.

  • What evidence supports attributing the question-invariant mean direction specifically to formatting, rather than to other shared model states or task-level computations? Would the main conclusions still hold under the more conservative interpretation that the statistic cannot distinguish these possibilities?

  • Could the authors provide an accessible, timestamped preregistration and distinguish the originally specified confirmatory analyses from subsequent exploratory analyses? In particular, which scaling-related statement should be regarded as the final conclusion, given the inconsistency between the abstract and Section 5.3?

Reviewer_u5FUOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper audits two diagnostics used to characterize reasoning in language models and argues that both report a different quantity than the one they are claimed to measure. The analysis uses Qwen3 models, which expose a single thinking switch while holding weights, prompt, decoding, and seed fixed. This isolates the effect of thinking from all other factors.

The first diagnostic is a variance share curve computed from the difference between thinking and non thinking hidden states. A drop in this curve is commonly read as evidence of rank structure or a phase transition. The paper shows that the leading direction of the difference is the direction shared across all questions. That direction reflects format, not question specific computation. A non thinking control produces the same drop, so the drop does not indicate a thinking related transition.

The second diagnostic measures binary agreement between a verbalized confidence and a probe, read as a measure of divergence. The paper shows that one side is almost always above the decision threshold. As a result, the agreement score reduces to the sign of the probe and does not measure divergence.

Both results share one mechanism. When one side of a constructed comparison is near constant, the comparison degenerates into a function of the other side.

Critical strengths

1/The central result is precise and verifiable. A single mechanism accounts for both failures, and the paper provides cheap checks that others can apply before relying on either diagnostic. 2/The experimental design is well controlled. The same weight single switch setup, byte identical prompts across scales, and the zero content prefix control isolate the intended effect. The non thinking control is the decisive test. 3/The analysis is disciplined about its own claims. It separates the supported claim (the leading direction) from the unsupported ones (monotonicity, scaling), applies a family wise test that rejects its own ordering, and documents retired wordings and revised readings. 4/On novelty: the paper extends prior work showing that probes read format rather than reasoning mode. It advances from identifying what the signal is not to characterizing what the summary statistic is. This is a substantive extension rather than a restatement.

Critical weaknesses

1/The scope is limited. The evidence rests on one model family, one mathematics dataset, and two diagnostics. The mechanism is general, but its practical impact is demonstrated only in this setting. The prevalence of the problem across other models, tasks, and diagnostics is not established. This is my major reason for weak reject. 2/The positive contribution is limited. After the two negative results, what remains is a crossing position that marks the end of shared format and a late segment that is reproducible but not shown to carry semantic content. The paper does not propose a validated replacement statistic, so the contribution is primarily diagnostic.

Questions for authors

1/Does the mechanism appear outside Qwen3 and outside MATH500? One additional model family or one non mathematics task would indicate whether the problem is structural. 2/For the first diagnostic, is there a corrected statistic that tracks rank structure once the shared direction is removed? A validated alternative would convert the critique into a usable tool. 3/Can any test separate reproducible structure in the late segment from genuine semantic content, rather than leaving the distinction open? 4/The agreement diagnostic fails when one estimator saturates. Would the same estimators become informative on a task where verbalized confidence varies, as the smallest model suggests?

Reviewer_mtn5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits two standard diagnostics on same-weight Qwen3 think/no-think pairs. The authors hold model weights, prompts, seeds, and greedy decoding fixed while toggling enable thinking, and introduce c1–c6 controlled continuations. Main experiments are performed on MATH500 with Qwen3-0.6B through 32B.

Critical strengths

The paper conducts a range of statistical analyses for diagnosis and attribution, including effective-rank and Spearman correlation analyses. The carefully matched experimental settings also provide reasonably reliable ablation results.

Critical weaknesses
  1. Limited novelty. Although the paper presents extensive experiments analyzing the same-weight dual-mode model design with an enable_thinking switch, the overall contribution remains closer to an engineering-oriented empirical study than to a substantial methodological advance. Consequently, the broader insights offered to the community appear somewhat limited. The authors could strengthen the paper by clarifying the underlying principles revealed by these analyses and deriving more generalizable conclusions.

  2. Limited generalizability. The current evaluation focuses primarily on the Qwen3 model family and a single text-only mathematical reasoning benchmark, MATH500. Other model families and task types are not considered, making it difficult to determine whether the reported findings generalize beyond this specific setting. To better establish the robustness and generality of the conclusions, the authors should consider evaluating the method across additional model families and a broader range of benchmarks, including tasks beyond mathematical reasoning.

Questions for authors

See Weakness part

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.