- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CUBE, a framework for evaluating sample-level reasoning quality in large language models (LLMs) by analyzing the joint geometric dynamics of all tokens across Transformer layers. Specifically, CUBE learns a shared, low-dimensional, outcome-discriminative coordinate space through supervised calibration and characterizes reasoning trajectories using three interpretable metrics: Detour Ratio, Convergence Ratio, and Early-Step Consistency. Experiments across multiple model families and scales demonstrate that these geometric metrics are informative about answer correctness and that the selected training subsets improve downstream accuracy compared with existing selection baselines.
Critical strengths
- CUBE jointly considers the representation changes of all tokens across different Transformer layers, rather than tracking individual tokens or final-layer representations alone. It summarizes these complex internal dynamics using three interpretable geometric metrics.
- The paper provides extensive experimental evaluations across mathematical reasoning, code generation, and multi-hop question answering tasks. It also examines the effectiveness of the proposed geometric metrics across different model families and reasoning modes.
Critical weaknesses
- The proposed method assumes that higher-quality reasoning is generally associated with more direct changes in internal representations, clearer stabilization patterns, and more consistent early updates. Although this assumption is intuitively reasonable, it has not been sufficiently validated.
- The authors treat the hidden states of all tokens across Transformer layers as a reasoning trajectory and use the analogy of solving a Rubik's Cube to interpret layer-wise changes as successive refinements of the reasoning process. However, feature transformations across Transformer layers are not equivalent to step-by-step decisions in natural-language reasoning.
- CUBE-EVAL evaluates the reasoning quality of a given sample by analyzing the layer-wise geometric dynamics of its internal representations. However, the experimental texts are generated by DeepSeek-V4-Pro by rewriting questions and their reference solutions. Since these texts already contain the correct answers, they differ substantially from reasoning processes generated independently by the evaluated models.
- The authors introduce three metrics—Detour Ratio, Convergence Ratio, and Early-Step Consistency—to measure trajectory directness, stabilization, and consistency of early updates, respectively. However, further justification is needed to explain why these three metrics together can adequately characterize reasoning quality.
Questions for authors
See weakness
5 comments
CUBE-EVAL evaluates the reasoning quality of a given sample by analyzing the layer-wise geometric dynamics of its internal representations. However, the experimental texts are generated by DeepSeek-V4-Pro by rewriting questions and their reference solutions. Since these texts already contain the correct answers, they differ substantially from reasoning processes generated independently by the evaluated models.
W3. Thank you very much for this comment. We would like to clarify the scope of this concern: it applies to CUBE-EVAL only, and there the use of supplied texts is a deliberate design choice. CUBE-EVAL is an evaluator for supplied texts: as Section 4.2.2 states, "because the paragraph already states the answer, CUBE-EVAL measures how the model processes a given solution rather than whether it can find one." Using a fixed text is what makes the comparison between models valid. The corpus is frozen across evaluated models, so all models read identical inputs, and differences in the score come from the model and not from the text each model would have written. The scorer is also calibrated on the model's own reasoning: the projection B is fitted on the trajectories the model generated when solving the benchmark questions, and is then applied to the fixed paragraphs (Appendix A.4).
Reasoning generated independently by the evaluated models is covered by the other experiments in the paper. For Table 1, each benchmark question elicits "a reasoning trace and final answer, while the token-wise, layer-wise hidden states and binary correctness label are cached during generation" (Section 4.2.1), and CUBE-SELECTOR scores "the trajectory generated while the strong model solves each OpenR1 problem" (Appendix A.4). The CUBE-EVAL score is also consistent with the models' own performance: across the eight model–benchmark cells, it has a positive rank correlation with accuracy on the original benchmarks (Spearman ρ = 0.55, Figure 3).
We add an item-level test of this relation (Table R5). For each item, we take the three signatures a model produces while reading the fixed paragraph and use them to predict whether the same model answers the original benchmark question correctly on its own.
Table R5. Predicting a model's own correctness on the original question from the signatures it produces while reading the fixed CUBE-EVAL paragraph. AUROC with 95% CIs, for a random-forest and a logistic-regression scorer.
| Model | Domain | N | Correct / Incorrect | Random forest | Logistic regression |
|---|---|---|---|---|---|
| Qwen3-8B | Math | 530 | 500 / 30 | 0.756 [0.646, 0.863] | 0.733 [0.614, 0.840] |
| Llama-3.1-8B | Math | 500 | 230 / 270 | 0.565 [0.513, 0.614] | 0.564 [0.513, 0.616] |
| Qwen3-8B | Multi-hop QA | 610 | 60 / 550 | 0.514 [0.438, 0.584] | 0.649 [0.564, 0.722] |
| Llama-3.1-8B | Multi-hop QA | 610 | 36 / 574 | 0.554 [0.459, 0.643] | 0.555 [0.455, 0.646] |
On mathematics, the reading geometry predicts the model's own correctness for both models and both scorers (0.756 for Qwen3-8B and 0.565 for Llama-3.1-8B with the random forest used in the paper). The geometry a model shows on a supplied mathematical solution is therefore related to its own ability to solve that problem. The multi-hop QA results are listed in the same table. We will add Table R5 to the appendix and state the scope of CUBE-EVAL explicitly where it is first introduced.
The authors introduce three metrics—Detour Ratio, Convergence Ratio, and Early-Step Consistency—to measure trajectory directness, stabilization, and consistency of early updates, respectively. However, further justification is needed to explain why these three metrics together can adequately characterize reasoning quality.
W4. Thank you very much for this comment. The three signatures are chosen to describe three different properties of a layer trajectory, and the paper supports this choice both formally and empirically. We would also like to clarify the scope of the claim: the paper uses reasoning quality "as an operational term for three properties of the internal process—directness, relative stabilization, and the consistency of early update scales" (Section 1), and each signature measures one of them.
Each signature measures something the other two do not. The Detour Ratio measures how much of the accumulated motion becomes net displacement (Definition 3.2). As Section 3.2 states, "Directness alone does not reveal whether the trajectory settles near the output", so the Convergence Ratio compares early-layer with late-layer motion (Definition 3.3). Early-Step Consistency measures how regular the early update scales are (Definition 3.4). Together they describe how far the trajectory moves in net terms, when the motion happens across depth, and how regular the early steps are. Under an idealized increment model, Theorem 3.1 shows that D increases with the directional signal-to-noise ratio of the layer updates and that C increases, relative to the model-wide depth baseline, under sample-specific contraction. F is invariant to the scale of the steps (Proposition D.2).
The experiments in the paper show that the three signatures carry different information. In Table 10, the most informative single signature differs across contrasts: F for Hallucinated vs. Correct (0.700), C for Hallucinated vs. Other incorrect (0.598) and D for Other incorrect vs. Correct (0.660). The combination of the three has the highest AUC in each contrast (0.720, 0.609 and 0.685). Table 9 shows what a single signature would miss: hallucinated responses are more direct than other incorrect responses (1/D of 2.405 vs. 2.459), but they show the strongest late-layer motion (1/C of 3.236) and the lowest Early-Step Consistency (0.645).
We add a leave-one-signature-out analysis (Table R6).
Table R6. Out-of-fold AUROC with all three signatures and with one signature removed. Bold marks the largest drop in each row. The projection is an unsupervised PCA and the signatures are computed on the layer trajectory of the final answer token, so absolute values are not directly comparable with Table 1.
| Model | Benchmark | D, C, F | without D | without C | without F |
|---|---|---|---|---|---|
| Llama-3.1-8B | MATH-500 | 0.689 | 0.632 | 0.685 | 0.674 |
| Hunyuan-7B | LiveCodeBench | 0.834 | 0.729 | 0.693 | 0.643 |
For Llama-3.1-8B on MATH-500, removing D lowers AUROC from 0.689 to 0.632, while removing C or F changes it much less (0.685 and 0.674). For Hunyuan-7B on LiveCodeBench, removing C costs the most (0.834 to 0.693), followed by D (0.729). The pairwise Spearman correlation between signatures does not exceed 0.72 in absolute value in any cell. We will add Table R6 to the appendix and add a short paragraph to Section 3.2 that states what each signature adds to the other two.
W1. Thank you very much for this comment. We would like to clarify that the paper does not take this relation as an assumption: it states it as an operationalization and tests it. As the introduction states, "We test the criterion validity of this operationalization through its associations with answer correctness and controlled failure types." The paper validates the relation in three ways, and we add a fourth here.
The new validation does not involve correctness (Table R3). In Appendix C.2, DeepSeek-V4-Pro rewrites one verified-correct solution per problem into a refined version and a deliberately detoured version. We compare the two, using only rewrites that are judged correct (2,153 refined and 2,990 detoured), with cross-validation grouped by problem.
Table R3. Refined versus detoured rewrites, all judged correct. Out-of-fold AUROC with five folds grouped by problem. Length is the log token count.
The signatures separate refined from detoured solutions with AUROC 0.664, and with 0.561 (95% CI [0.526, 0.596]) on 553 pairs matched within ±10% in token count. Since both groups are correct, this separation cannot come from the outcome label. We will add Table R3 to the appendix, and we will refer to these validations in Section 3.2, where the three properties are introduced.
W2. Thank you very much for this comment. We would like to clarify a misunderstanding: CUBE does not equate layer-wise transformations with step-by-step decisions in natural-language reasoning. It measures a different axis. The introduction separates two axes along which prior work measures evolution: token-wise, along the generated chain of thought, and depth-wise, across layers. CUBE is depth-wise. As Appendix B.1 states, "this is depth-wise rather than token-wise dynamics: all tokens form the state, but the evolution axis is layer depth." The natural-language reasoning steps lie along the token dimension, so they are part of the state that CUBE tracks. The trajectory describes how the model's representation of the whole solution is refined from one layer to the next. Treating depth as a refinement axis follows prior work: the Tuned Lens decodes each layer's intermediate prediction (Belrose et al., 2023), and intrinsic-dimension analyses show representations expanding and then compressing with depth (Joshi et al., 2025).
The Rubik's-cube analogy gives intuition for the three signatures. It maps one move to one layer ("viewing each layer as one move toward a solved configuration"), not to one sentence of the solution. The signatures themselves are defined only from layer-to-layer increments (Definitions 3.2–3.4), and whether they are informative is tested empirically (Table 1, Appendix C.2), not derived from the analogy.
We agree that the caption of Figure 2(a), "A reasoning step is a Rubik's-cube move", invites a token-wise reading. We will reword it to "a layer update" and move the sentence from Appendix B.1 into Section 3.1.