acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Sample Reasoning Quality from Internal Geometric Dynamics in LLMs

Abstract

Understanding and benchmarking how well a large language model reasons over a given sample is crucial, yet it is rarely legible from the model's surface output. Prior work overlooks how reasoning evolves across the model's layers, which we argue is indispensable to judging reasoning quality. Intuitively, just as a Rubik's cube can be solved by many strategies, the outcome is only one aspect of reasoning; whether the underlying strategy is concise and direct matters just as much. From this view, we propose Cube, a framework that benchmarks a sample's reasoning quality directly from internal geometric dynamics. By tracking the reasoning trajectory in a low-dimensional reasoning subspace, Cube identifies the characteristic cross-layer transformations and summarizes them with geometric signatures. Cube has broad applications, and in this work we present Cube-Eval and Cube-Selector for demonstration: one assesses reasoning quality directly from the model's internal dynamics, without any ground-truth labels, and the other pinpoints samples of low reasoning quality for targeted training. Extensive experiments demonstrate the effectiveness of both Cube-Eval and Cube-Selector.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, large language models, reasoning data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_kKFyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CUBE, a framework for evaluating sample-level reasoning quality in large language models (LLMs) by analyzing the joint geometric dynamics of all tokens across Transformer layers. Specifically, CUBE learns a shared, low-dimensional, outcome-discriminative coordinate space through supervised calibration and characterizes reasoning trajectories using three interpretable metrics: Detour Ratio, Convergence Ratio, and Early-Step Consistency. Experiments across multiple model families and scales demonstrate that these geometric metrics are informative about answer correctness and that the selected training subsets improve downstream accuracy compared with existing selection baselines.

Critical strengths
  1. CUBE jointly considers the representation changes of all tokens across different Transformer layers, rather than tracking individual tokens or final-layer representations alone. It summarizes these complex internal dynamics using three interpretable geometric metrics.
  2. The paper provides extensive experimental evaluations across mathematical reasoning, code generation, and multi-hop question answering tasks. It also examines the effectiveness of the proposed geometric metrics across different model families and reasoning modes.
Critical weaknesses
  1. The proposed method assumes that higher-quality reasoning is generally associated with more direct changes in internal representations, clearer stabilization patterns, and more consistent early updates. Although this assumption is intuitively reasonable, it has not been sufficiently validated.
  2. The authors treat the hidden states of all tokens across Transformer layers as a reasoning trajectory and use the analogy of solving a Rubik's Cube to interpret layer-wise changes as successive refinements of the reasoning process. However, feature transformations across Transformer layers are not equivalent to step-by-step decisions in natural-language reasoning.
  3. CUBE-EVAL evaluates the reasoning quality of a given sample by analyzing the layer-wise geometric dynamics of its internal representations. However, the experimental texts are generated by DeepSeek-V4-Pro by rewriting questions and their reference solutions. Since these texts already contain the correct answers, they differ substantially from reasoning processes generated independently by the evaluated models.
  4. The authors introduce three metrics—Detour Ratio, Convergence Ratio, and Early-Step Consistency—to measure trajectory directness, stabilization, and consistency of early updates, respectively. However, further justification is needed to explain why these three metrics together can adequately characterize reasoning quality.
Questions for authors

See weakness

5 comments

The proposed method assumes that higher-quality reasoning is generally associated with more direct changes in internal representations, clearer stabilization patterns, and more consistent early updates. Although this assumption is intuitively reasonable, it has not been sufficiently validated.

W1. Thank you very much for this comment. We would like to clarify that the paper does not take this relation as an assumption: it states it as an operationalization and tests it. As the introduction states, "We test the criterion validity of this operationalization through its associations with answer correctness and controlled failure types." The paper validates the relation in three ways, and we add a fourth here.

  • Correctness of the model's own solutions (Table 1). The three signatures predict correctness with AUC above 0.5 in all 15 model–domain cells (0.520 to 0.882). 42 of the 45 coefficients are positive, so correct solutions are the more direct, earlier-stabilizing and more consistent ones.
  • Controlled response types (Table 9). Correct responses are the most direct (1/D of 2.257 against 2.405 and 2.459), show the least late-layer motion relative to early motion (1/C of 3.061 against 3.236 and 3.104), and have the highest Early-Step Consistency (0.666 against 0.645 and 0.652).
  • Downstream use (Table 2). Training data selected with these signatures gives higher accuracy than the strongest comparator in all six model–benchmark cells, by 1.1 points for Qwen3-8B and 1.0 point for Hunyuan-7B on average.

The new validation does not involve correctness (Table R3). In Appendix C.2, DeepSeek-V4-Pro rewrites one verified-correct solution per problem into a refined version and a deliberately detoured version. We compare the two, using only rewrites that are judged correct (2,153 refined and 2,990 detoured), with cross-validation grouped by problem.

Table R3. Refined versus detoured rewrites, all judged correct. Out-of-fold AUROC with five folds grouped by problem. Length is the log token count.

FeaturesAUROC95% CI
D, C, F jointly0.664[0.649, 0.678]
Detour Ratio (D) alone0.501[0.485, 0.517]
Convergence Ratio (C) alone0.640[0.625, 0.655]
Early-Step Consistency (F) alone0.465[0.448, 0.480]
Length alone0.825[0.810, 0.840]
Length + D, C, F0.859[0.848, 0.871]
D, C, F jointly, 553 length-matched pairs0.561[0.526, 0.596]

The signatures separate refined from detoured solutions with AUROC 0.664, and with 0.561 (95% CI [0.526, 0.596]) on 553 pairs matched within ±10% in token count. Since both groups are correct, this separation cannot come from the outcome label. We will add Table R3 to the appendix, and we will refer to these validations in Section 3.2, where the three properties are introduced.

The authors treat the hidden states of all tokens across Transformer layers as a reasoning trajectory and use the analogy of solving a Rubik's Cube to interpret layer-wise changes as successive refinements of the reasoning process. However, feature transformations across Transformer layers are not equivalent to step-by-step decisions in natural-language reasoning.

W2. Thank you very much for this comment. We would like to clarify a misunderstanding: CUBE does not equate layer-wise transformations with step-by-step decisions in natural-language reasoning. It measures a different axis. The introduction separates two axes along which prior work measures evolution: token-wise, along the generated chain of thought, and depth-wise, across layers. CUBE is depth-wise. As Appendix B.1 states, "this is depth-wise rather than token-wise dynamics: all tokens form the state, but the evolution axis is layer depth." The natural-language reasoning steps lie along the token dimension, so they are part of the state that CUBE tracks. The trajectory describes how the model's representation of the whole solution is refined from one layer to the next. Treating depth as a refinement axis follows prior work: the Tuned Lens decodes each layer's intermediate prediction (Belrose et al., 2023), and intrinsic-dimension analyses show representations expanding and then compressing with depth (Joshi et al., 2025).

The Rubik's-cube analogy gives intuition for the three signatures. It maps one move to one layer ("viewing each layer as one move toward a solved configuration"), not to one sentence of the solution. The signatures themselves are defined only from layer-to-layer increments (Definitions 3.2–3.4), and whether they are informative is tested empirically (Table 1, Appendix C.2), not derived from the analogy.

We agree that the caption of Figure 2(a), "A reasoning step is a Rubik's-cube move", invites a token-wise reading. We will reword it to "a layer update" and move the sentence from Appendix B.1 into Section 3.1.

CUBE-EVAL evaluates the reasoning quality of a given sample by analyzing the layer-wise geometric dynamics of its internal representations. However, the experimental texts are generated by DeepSeek-V4-Pro by rewriting questions and their reference solutions. Since these texts already contain the correct answers, they differ substantially from reasoning processes generated independently by the evaluated models.

W3. Thank you very much for this comment. We would like to clarify the scope of this concern: it applies to CUBE-EVAL only, and there the use of supplied texts is a deliberate design choice. CUBE-EVAL is an evaluator for supplied texts: as Section 4.2.2 states, "because the paragraph already states the answer, CUBE-EVAL measures how the model processes a given solution rather than whether it can find one." Using a fixed text is what makes the comparison between models valid. The corpus is frozen across evaluated models, so all models read identical inputs, and differences in the score come from the model and not from the text each model would have written. The scorer is also calibrated on the model's own reasoning: the projection B is fitted on the trajectories the model generated when solving the benchmark questions, and is then applied to the fixed paragraphs (Appendix A.4).

Reasoning generated independently by the evaluated models is covered by the other experiments in the paper. For Table 1, each benchmark question elicits "a reasoning trace and final answer, while the token-wise, layer-wise hidden states and binary correctness label are cached during generation" (Section 4.2.1), and CUBE-SELECTOR scores "the trajectory generated while the strong model solves each OpenR1 problem" (Appendix A.4). The CUBE-EVAL score is also consistent with the models' own performance: across the eight model–benchmark cells, it has a positive rank correlation with accuracy on the original benchmarks (Spearman ρ = 0.55, Figure 3).

We add an item-level test of this relation (Table R5). For each item, we take the three signatures a model produces while reading the fixed paragraph and use them to predict whether the same model answers the original benchmark question correctly on its own.

Table R5. Predicting a model's own correctness on the original question from the signatures it produces while reading the fixed CUBE-EVAL paragraph. AUROC with 95% CIs, for a random-forest and a logistic-regression scorer.

ModelDomainNCorrect / IncorrectRandom forestLogistic regression
Qwen3-8BMath530500 / 300.756 [0.646, 0.863]0.733 [0.614, 0.840]
Llama-3.1-8BMath500230 / 2700.565 [0.513, 0.614]0.564 [0.513, 0.616]
Qwen3-8BMulti-hop QA61060 / 5500.514 [0.438, 0.584]0.649 [0.564, 0.722]
Llama-3.1-8BMulti-hop QA61036 / 5740.554 [0.459, 0.643]0.555 [0.455, 0.646]

On mathematics, the reading geometry predicts the model's own correctness for both models and both scorers (0.756 for Qwen3-8B and 0.565 for Llama-3.1-8B with the random forest used in the paper). The geometry a model shows on a supplied mathematical solution is therefore related to its own ability to solve that problem. The multi-hop QA results are listed in the same table. We will add Table R5 to the appendix and state the scope of CUBE-EVAL explicitly where it is first introduced.

The authors introduce three metrics—Detour Ratio, Convergence Ratio, and Early-Step Consistency—to measure trajectory directness, stabilization, and consistency of early updates, respectively. However, further justification is needed to explain why these three metrics together can adequately characterize reasoning quality.

W4. Thank you very much for this comment. The three signatures are chosen to describe three different properties of a layer trajectory, and the paper supports this choice both formally and empirically. We would also like to clarify the scope of the claim: the paper uses reasoning quality "as an operational term for three properties of the internal process—directness, relative stabilization, and the consistency of early update scales" (Section 1), and each signature measures one of them.

Each signature measures something the other two do not. The Detour Ratio measures how much of the accumulated motion becomes net displacement (Definition 3.2). As Section 3.2 states, "Directness alone does not reveal whether the trajectory settles near the output", so the Convergence Ratio compares early-layer with late-layer motion (Definition 3.3). Early-Step Consistency measures how regular the early update scales are (Definition 3.4). Together they describe how far the trajectory moves in net terms, when the motion happens across depth, and how regular the early steps are. Under an idealized increment model, Theorem 3.1 shows that D increases with the directional signal-to-noise ratio of the layer updates and that C increases, relative to the model-wide depth baseline, under sample-specific contraction. F is invariant to the scale of the steps (Proposition D.2).

The experiments in the paper show that the three signatures carry different information. In Table 10, the most informative single signature differs across contrasts: F for Hallucinated vs. Correct (0.700), C for Hallucinated vs. Other incorrect (0.598) and D for Other incorrect vs. Correct (0.660). The combination of the three has the highest AUC in each contrast (0.720, 0.609 and 0.685). Table 9 shows what a single signature would miss: hallucinated responses are more direct than other incorrect responses (1/D of 2.405 vs. 2.459), but they show the strongest late-layer motion (1/C of 3.236) and the lowest Early-Step Consistency (0.645).

We add a leave-one-signature-out analysis (Table R6).

Table R6. Out-of-fold AUROC with all three signatures and with one signature removed. Bold marks the largest drop in each row. The projection is an unsupervised PCA and the signatures are computed on the layer trajectory of the final answer token, so absolute values are not directly comparable with Table 1.

ModelBenchmarkD, C, Fwithout Dwithout Cwithout F
Llama-3.1-8BMATH-5000.6890.6320.6850.674
Hunyuan-7BLiveCodeBench0.8340.7290.6930.643

For Llama-3.1-8B on MATH-500, removing D lowers AUROC from 0.689 to 0.632, while removing C or F changes it much less (0.685 and 0.674). For Hunyuan-7B on LiveCodeBench, removing C costs the most (0.834 to 0.693), followed by D (0.729). The pairwise Spearman correlation between signatures does not exceed 0.72 in absolute value in any cell. We will add Table R6 to the appendix and add a short paragraph to Section 3.2 that states what each signature adds to the other two.

Reviewer_qpdcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CUBE, which learns an outcome-supervised low-dimensional projection of hidden states expressed through the model's output map. It summarizes the resulting layerwise trajectories with Detour Index, Convergence Ratio, and Early-Step Consistency. CUBE-EVAL studies the association of these signatures with the correctness of supplied solutions, while CUBE-SELECTOR uses a stronger model's geometry to choose a fixed-size post-training set. The paper reports predictive results across models and domains, idealized geometric analysis, and downstream improvements from data selection.

Critical strengths
  1. The geometric signatures are compact and relatively easy to interpret. They provide an accessible description of how hidden states change across layers.
  2. The paper evaluates the approach across models and domains, giving a broader picture than a single-task demonstration.
  3. The fixed-size data-selection experiment is a useful practical application. It goes beyond measuring correlations and tests whether the proposed scores can help with post-training data selection.
Critical weaknesses
  1. The central interpretation as reasoning quality is not sufficiently supported. Since the projection is trained using outcome labels, a relationship with correctness is not especially surprising. The paper does not convincingly show that the resulting geometry captures the quality of the reasoning beyond its final outcome.
  2. The comparisons leave the incremental contribution unclear. In particular, stronger supervised baselines on terminal or pooled hidden states are needed to establish the value of the trajectory signatures. Comparisons with unsupervised geometry do not isolate how much of the improvement comes simply from learning the projection with labels.
  3. I have concerns about the evaluation protocol and the strength of the statistical claims. The projection dimension is selected using examples that later reappear in cross-validation, and the fold-wise significance analysis does not adequately address dependence between the fitted estimates. These issues reduce confidence in the reported effects, especially where predictive performance is already modest.
Questions for authors
  1. How does CUBE compare with supervised probes on terminal or pooled hidden states under matched training and tuning conditions?
  2. Do the results hold when hyperparameter selection is kept separate from evaluation, and when dependence between folds is accounted for?
  3. Can the signatures distinguish better and worse reasoning among solutions that are all correct, with source and length controlled?
4 comments

How does CUBE compare with supervised probes on terminal or pooled hidden states under matched training and tuning conditions?

Q1. Thank you for suggesting this comparison. We ran this comparison under matched conditions (Table R1). We trained supervised probes on the terminal hidden state and on the pooled hidden states. The probes use the same correctness labels and the same outer folds as the three signatures, and they are tuned by inner cross-validation. All numbers are out of fold.

Table R1. Out-of-fold AUROC for correctness prediction, averaged over Qwen3-8B, Llama-3.1-8B and Hunyuan-7B. Bold marks the best method in each domain.

MethodMathCodeQA
Three signatures (D, C, F)0.7590.8870.573
Supervised probe, terminal hidden state0.5690.5740.531
Supervised probe, pooled hidden states0.5470.5630.506

The three signatures have the highest AUROC in all three domains. They exceed the terminal-state probe by 0.190 on mathematics, 0.313 on code and 0.042 on QA, and the pooled-state probe by 0.212, 0.324 and 0.067. The margin is smallest on QA.

The signatures also describe the trajectory, which a probe's weight vector does not. Table 9 is an example: hallucinated responses are more direct than other incorrect responses but show the strongest late-layer motion. We note that Section 4.2.1 uses correctness prediction "to test whether CUBE's geometry carries information about reasoning correctness", and the paper does not propose the signatures as a correctness detector. We will add Table R1 to the appendix.

Do the results hold when hyperparameter selection is kept separate from evaluation, and when dependence between folds is accounted for?

Q2. Thank you for this feedback. The results hold under both checks.

Selection of k. Fixing k = 16 is not an optimistic choice, for three reasons. (i) As described in Appendix A.4, k was selected on a tuning subset of a single configuration (Qwen3-8B Think). The other 12 of the 15 cells in Table 1 contributed no trajectories or labels to that selection. (ii) The 3 cells that did contribute are the weakest in their domains (0.520 / 0.643 / 0.537), so the conclusions of Table 1 do not rest on them. (iii) When k is selected strictly inside each outer training fold from {4, 8, 16}, so that selection never sees evaluation data, AUROC changes by a median of +0.008 relative to fixed k = 16 (Table R4, 9 cells), and k = 16 is the best fixed value in only 4 of the 9 cells.

Table R4. Out-of-fold AUROC of the three signatures for fixed k and for k selected inside each outer training fold from {4, 8, 16}. Δ is the change relative to fixed k = 16, computed before rounding. Bold marks the best fixed k in each row.

ModelDomaink = 4k = 8k = 16k selected in training foldsΔ vs. k = 16
Qwen3-8BMath0.9160.9080.9010.908+0.007
Qwen3-8BCode0.6780.7560.7380.687−0.051
Qwen3-8BQA0.5700.5810.5920.614+0.022
Llama-3.1-8BMath0.6640.6900.7220.719−0.003
Llama-3.1-8BCode0.4710.6330.5400.592+0.052
Llama-3.1-8BQA0.6570.6670.6870.694+0.008
Hunyuan-7BMath0.7840.7570.5670.771+0.204
Hunyuan-7BCode0.7880.8350.8600.860+0.000
Hunyuan-7BQA0.5930.5660.5880.599+0.010

Dependence between folds. We understand the concern that the fold-wise estimates share training data. When this dependence is accounted for, the results agree with Table 1. Table 1 tests whether the three signatures predict correctness on held-out samples, and Section 4.2.1 concludes that the out-of-fold AUROC is above chance in every cell, with a strength that varies across models and domains. We re-evaluated this with two tests that do not treat folds as independent samples (Table R7): bootstrap 95% CIs, and a label-permutation test that refits the whole pipeline. In all 11 cells, AUROC is above 0.5 (0.516 to 0.868), and in 9 of them it is significant under both tests (CI excludes 0.5, permutation p ≤ 0.02). The other two are FanOutQA cells with only 17 and 21 correct answers out of 310, where the intervals are wide. We will add Tables R4 and R7 to the appendix.

Table R7. Out-of-fold AUROC of the three signatures per model and benchmark, with bootstrap 95% CIs and label-permutation p-values (100 permutations, whole pipeline refitted). A cell passes if the CI excludes 0.5 and p < 0.05. The projection is an unsupervised PCA (k = 16) and the signatures are computed on the layer trajectory of the final answer token, so absolute values are not directly comparable with Table 1.

ModelBenchmarkNCorrect / IncorrectAUROC95% CIPermutation pPass
Qwen3-8BMATH-500500479 / 210.868[0.792, 0.932]0.0099Yes
Qwen3-8BLiveCodeBench40020 / 3800.710[0.616, 0.796]0.0099Yes
Qwen3-8BMoreHopQA1118166 / 9520.582[0.536, 0.627]0.0099Yes
Qwen3-8BFanOutQA31017 / 2930.516[0.366, 0.662]0.3168No
Llama-3.1-8BMATH-500500230 / 2700.689[0.642, 0.735]0.0099Yes
Llama-3.1-8BLiveCodeBench40041 / 3590.626[0.526, 0.720]0.0198Yes
Llama-3.1-8BMoreHopQA111871 / 10470.644[0.572, 0.713]0.0099Yes
Llama-3.1-8BFanOutQA31021 / 2890.616[0.491, 0.740]0.0594No
Hunyuan-7BMATH-500500459 / 410.763[0.658, 0.859]0.0099Yes
Hunyuan-7BLiveCodeBench40045 / 3550.834[0.777, 0.884]0.0099Yes
Hunyuan-7BMoreHopQA1118102 / 10160.586[0.538, 0.636]0.0198Yes

Can the signatures distinguish better and worse reasoning among solutions that are all correct, with source and length controlled?

Q3. Thank you for this question. Yes, they can. Appendix C.2 already contains the data for this test: DeepSeek-V4-Pro rewrites one verified-correct solution per problem into a refined version and a deliberately detoured version. We merged the two into the Correct group, so this comparison was not reported in the submission. Table R3 reports it, using only rewrites that are judged correct (2,153 refined and 2,990 detoured). Cross-validation is grouped by problem, and the projection is fitted without access to the refined/detoured distinction.

Table R3. Refined versus detoured rewrites, all judged correct. Out-of-fold AUROC with five folds grouped by problem. Length is the log token count.

FeaturesAUROC95% CI
D, C, F jointly0.664[0.649, 0.678]
Detour Ratio (D) alone0.501[0.485, 0.517]
Convergence Ratio (C) alone0.640[0.625, 0.655]
Early-Step Consistency (F) alone0.465[0.448, 0.480]
Length alone0.825[0.810, 0.840]
Length + D, C, F0.859[0.848, 0.871]
D, C, F jointly, 553 length-matched pairs0.561[0.526, 0.596]

The three signatures separate refined from detoured solutions with AUROC 0.664. The two groups also differ in length, so we control for it in two ways. Adding the signatures to a length-only model raises AUROC from 0.825 to 0.859 (+0.035, 95% CI [0.027, 0.042]). On 553 refined–detoured pairs matched within ±10% in token count, the signatures alone reach 0.561 (95% CI [0.526, 0.596]). Since both groups are correct, this separation cannot come from the outcome label. We will add Table R3 to the appendix.

Thank you for the detailed response and the additional experiments. Comparing the three geometric signatures with supervised probes on the same folds addressed my concern about the baselines, and the nested selection of kk and full-pipeline permutation tests strengthened my confidence in the statistical results. I especially appreciated the comparison of length-matched, correct solutions, which showed that the signatures retain a modest ability to distinguish refined solutions from those with deliberate detours. These analyses address my main concerns, and I will raise my score accordingly.

Comments on the paper

7 comments
Area_Chair_D2U9OpenReview ↗

Emergency Comments

Summary

CUBE judges reasoning quality from a model's layer-by-layer internal trajectory rather than from its final answer: token states are projected into a shared, correctness-supervised space and summarized by three scalar signatures. CUBE-EVAL scores a supplied solution; CUBE-SELECTOR uses the same signatures to choose which problems to fine-tune on.

Strengths

1.Leakage control is clean.​ The projection, the standardizer, and the downstream predictor are all fit inside each fold and then frozen, and the calibration pool excludes the sample being scored​ — so a scored solution never touches its own answer. It's the easiest place to get burned, and they don't.

2.The scope claims are honest.​ The paper says "operational term" and claims only "criterion validity" — it never overreaches to "faithful reasoning." The Limitations go further and concede, in plain terms, that the failure-type analysis "does not compare quality among correct solutions."

Weakness

  1. The title's construct is never tested where it counts — between two correct answers. As I read it, no experiment ranks quality among all-right solutions. The appendix builds just that pair (a clean solve and a meandering one), then merges them into one "correct" class — and the Limitations concede it. What's left is coarse: right vs. wrong, informative but uneven, and partly carried by style (failures are rewrites, successes are originals) — precisely the confound the merge hides. The one implicit peek inside strong, mostly-correct reasoning (their best reasoning-tuned model) nearly flatlines (thinnest on math/multi-hop, a pulse on code). That weakens, not refutes, the "sees past the answer" claim — the central comparison is still unmade.

  2. Two abstract claims outrun the evidence.​ (i) "Families and scales": the cross-model sweep reports accuracy, not geometry (set "aside" per the paper), so no scaling result on the signatures exists — at most one strong/weak pair, a tier gap, not a scale curve. (ii) "Remain informative" is too uniform: of 15 cells, a few are clearly above chance, a broad middle merely moderate, a couple at chance. Say informative in some regimes.

  3. Novel but modest, and unpriced.​ The three signatures are classical trajectory readings; the theory interprets rather than predicts. The new part is the across-layer framing + data selection. But Table 1 never pits that interpretability against the obvious rival — a supervised readout of the same states, uncompressed — so we can't tell how much accuracy "looking inside" costs. Nor does it report whether the three signatures are three views or mostly one.

  4. Directionally consistent, not established.​ CUBE leads its main baseline in all six cells — unlikely by luck, but the cells share models/benchmarks, so that comfort is partly illusory. No seed spread or significance test; a single-pass wobble looks about the size of the gains. Fairly, same-items scoring means a paired test could tighten my bound — hence I ask for per-seed results, not call it noise. Decisive control missing: select by "the strong model just gets these right." Until then, geometry-selection ≠ difficulty-selection.

  5. The summary evidence is thin — the paper half-agrees.​ Its one aggregate view is a ranking check: two far-apart models × four benchmarks, one correlation on a handful of non-independent points (same two models reused), so significance is only order-of-magnitude — and two models alone could fake the trend via ability tier. Bonus caveat: the scored passage already contains the answer, a friendlier setup than the open-ended claim.

The title's construct is never tested where it counts — between two correct answers. As I read it, no experiment ranks quality among all-right solutions. The appendix builds just that pair (a clean solve and a meandering one), then merges them into one "correct" class — and the Limitations concede it. What's left is coarse: right vs. wrong, informative but uneven, and partly carried by style (failures are rewrites, successes are originals) — precisely the confound the merge hides. The one implicit peek inside strong, mostly-correct reasoning (their best reasoning-tuned model) nearly flatlines (thinnest on math/multi-hop, a pulse on code). That weakens, not refutes, the "sees past the answer" claim — the central comparison is still unmade.

W1. Thank you for this careful reading. This is the comparison that matters most for the paper's claim, and we have now made it. As you note, Appendix C.2 already contains the pair: DeepSeek-V4-Pro rewrites one verified-correct solution per problem into a refined version and a deliberately detoured version, and we merged the two into the Correct group. Table R3 compares them directly, using only rewrites that are judged correct (2,153 refined and 2,990 detoured). Cross-validation is grouped by problem, and the projection is fitted without access to the refined/detoured distinction.

Table R3. Refined versus detoured rewrites, all judged correct. Out-of-fold AUROC with five folds grouped by problem. Length is the log token count.

FeaturesAUROC95% CI
D, C, F jointly0.664[0.649, 0.678]
Detour Ratio (D) alone0.501[0.485, 0.517]
Convergence Ratio (C) alone0.640[0.625, 0.655]
Early-Step Consistency (F) alone0.465[0.448, 0.480]
Length alone0.825[0.810, 0.840]
Length + D, C, F0.859[0.848, 0.871]
D, C, F jointly, 553 length-matched pairs0.561[0.526, 0.596]

The three signatures separate refined from detoured solutions with AUROC 0.664. The two groups also differ in length, so we control for it in two ways. Adding the signatures to a length-only model raises AUROC from 0.825 to 0.859 (+0.035, 95% CI [0.027, 0.042]). On 553 refined–detoured pairs matched within ±10% in token count, the signatures alone reach 0.561 (95% CI [0.526, 0.596]). Both groups are correct, and both are rewrites by the same model, so this separation cannot come from the outcome label or from a difference between rewrites and originals.

We would also like to clarify where the style confound applies. It concerns the three-way comparison in Appendix C.2, where the Correct group includes unchanged source solutions and the two failure groups contain rewrites. It does not apply to Table 1, which is the main correct-versus-incorrect evidence: there, correct and incorrect solutions are both generated by the evaluated model itself in response to the benchmark questions (Section 4.2.1). Within Appendix C.2, the Hallucinated vs. Other incorrect contrast compares two groups of rewrites, and the signatures separate them with AUC 0.609 ± 0.015 (Table 10).

On Qwen3-8B in thinking mode: these are the weakest cells of Table 1 (0.520 on mathematics, 0.537 on multi-hop QA and 0.643 on code), and Section 4.2.1 reports this dependence on inference mode. That row is still a correct-versus-incorrect test. The comparison among correct solutions is the one in Table R3. We will add Table R3 to the paper and revise the corresponding sentence in the Limitations.

Two abstract claims outrun the evidence. (i) "Families and scales": the cross-model sweep reports accuracy, not geometry (set "aside" per the paper), so no scaling result on the signatures exists — at most one strong/weak pair, a tier gap, not a scale curve. (ii) "Remain informative" is too uniform: of 15 cells, a few are clearly above chance, a broad middle merely moderate, a couple at chance. Say informative in some regimes.

W2. Thank you for checking the abstract against the evidence. In both cases the main text makes the more precise statement, and we will bring the abstract in line with it.

(i) The signature results cover model families, not model sizes. Table 1 reports the signatures for four families (Qwen3-8B in two inference modes, Llama-3.1-8B, GLM-4-9B and Hunyuan-7B), and Table 2 and Appendix C.2 use a fifth model, DeepSeek-V4-Flash. The scaling check reports answer accuracy only: Section 4.1 states that "smaller checkpoints are used only in the scaling check", and Appendix C.3 ends with "geometric signatures are analyzed separately". The paper therefore contains no result on how the signatures change with model size, and "scales" in the abstract was imprecise. We will change the phrase to "Across multiple model families" and describe Figure 5 as an accuracy reference.

(ii) Section 4.2.1 already states the qualified version: "The joint signatures yield above-chance AUC point estimates in every reported cell, but the strength of the association is not consistent across settings", and it concludes that "Table 1 shows model-, domain-, and mode-dependent associations with correctness." In Table 1, AUC is above 0.5 in all 15 cells. It is at least 0.60 in 13 cells and at least 0.74 in 5. The two cells below 0.60 (0.520 and 0.537) are both Qwen3-8B in thinking mode. The abstract compressed this into "remain informative". We will revise the sentence to: "Across multiple model families, the signatures are informative about answer correctness, with a strength that depends on the model, the domain and the inference mode."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.