acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Reasoning Paths as Phase-Structured Trajectories

Abstract

Large language models often improve problem-solving performance by generating multi-step reasoning paths, yet how to analyze the hidden states along these paths remains unclear. Existing approaches typically assign each intermediate state the final-answer correctness label and train probes across heterogeneous questions. We argue that this protocol obscures reasoning dynamics in two ways: (1) correctness prediction can exploit question-level variation rather than path quality, and (2) states aligned by absolute step indices may correspond to different functional phases of reasoning. In this work, we propose to view reasoning paths as phase-structured trajectories within fixed questions. We instantiate this view as PAIR, short for Phase-Aligned Intra-question Reasoning. PAIR samples multiple trajectories for each question, maps variable-length paths into shared relative phases based on normalized trajectory progress, and compares successful and unsuccessful trajectories only within the same question and phase. This yields phase-specific path-quality directions that better isolate path-quality signals from question-level variation. Empirically, we find that standard across-question correctness probes lose much of their predictive power under within-question evaluation, suggesting that these probes partly rely on question-level information. PAIR improves within-question trajectory ranking and Best-of-\(N\) trajectory selection across models and benchmarks. Phase-wise steering further shows that the learned directions can change generation outcomes, providing causal evidence that they capture trajectory-relevant information.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Reasoning, Mechanistic Interpretability, Representation Analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ksnoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PAIR, a phase-aligned within-question framework for analyzing hidden states along reasoning trajectories. Instead of comparing states across different questions or aligning trajectories by absolute step indices, PAIR compares successful and unsuccessful trajectories for the same question at similar relative phases. The method learns phase-specific path-quality directions and evaluates them on trajectory ranking, Best-of-N selection, and activation steering. The results suggest that standard across-question probes partly capture question-level information, while PAIR provides more informative trajectory-level signals.

Critical strengths
  1. The paper identifies an important limitation of across-question correctness probing, showing that probe performance may partly reflect question-level information rather than reasoning quality.

  2. PAIR is well motivated and shows consistent gains in trajectory ranking, Best-of-N selection, and steering.

Critical weaknesses
  1. The paper is difficult to follow, and the main motivation and method could be explained more clearly.

  2. The contribution of phase alignment is not fully isolated. The strong normalized-step baseline suggests that part of the gain may come from relative-position normalization rather than meaningful phase structure.

Questions for authors
  1. Could the authors better isolate the benefit of phase alignment from simple relative-position normalization? The normalized-step baseline is already strong in several settings.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ksno (1/2): W1

Thank you for your constructive feedback. We hope our responses will strengthen confidence in supporting our submission.

W1. Clarity of motivation and method

Thank you for this feedback. To make the main idea easier to follow, we summarize it here and have revised the paper to make it clearer in the same way.

Motivation. Prior work probes hidden states along a reasoning trajectory by labeling every state with the final-answer correctness and training a probe across many questions. This has two problems. First, the probe can rely on how hard a question is rather than on the quality of a particular trajectory. Second, states at the same absolute step index can play different roles in different trajectories.

Method. PAIR addresses both problems. It samples many trajectories for each question and compares correct and incorrect trajectories only within the same question, which removes question-level differences. It also maps each trajectory onto a fixed number of phases by relative progress, so that compared states sit at similar points in their trajectories. Within each phase, it learns a direction that ranks correct trajectories above incorrect ones. We use these directions for trajectory ranking and Best-of-N selection, and test their causal effect with steering.

Revisions made in the updated manuscript.

  • The abstract and introduction now summarize the new results, the introduction states how comparable positions are approximated, and the contribution list is updated accordingly.
  • Sec. 3 now opens with a one-sentence summary of PAIR, and Sec. 3.2 states explicitly that phase alignment is an annotation-free approximation of functional alignment.
  • Sec. 4.3 now describes the two new variants in Table 2 and explains what each comparison isolates.
  • Sec. 4.4 now describes the offline steering protocol and separates the use of later phases for readout from the use of early phases for steering. Table 3 now covers all four models.
  • The appendix now describes the segmentation rule with its fallback (A.4.1), and adds a controlled comparison of phase alignment (A.7), the functional-stage analysis (A.8), and cross-task transfer results (A.9).

If there are any other parts that remain unclear, we would be grateful for any pointers and will revise them further.

AuthorsOpenReview ↗

Response to Reviewer ksno (2/2): W2 and Q1

W2 & Q1. Phase Alignment and Relative-Position Normalization

Thank you for this question. We would like to clarify that PAIR does use relative position, and the strong Normalized-step baseline confirms that relative position matters. Our results show that (1) PAIR's gain comes from combining relative position with within-question comparison, not from relative position alone, and (2) the resulting phases correspond to meaningful reasoning stages. Normalized-step takes the step closest to 80% progress in one greedy trajectory per question and trains a pointwise probe on it. PAIR instead compares correct and incorrect trajectories of the same question at each phase. To separate these components, we compare four methods from the revised Table 2 (Section 4.3):

ModelMethodWQ-AUC (GSM8K)BoN@32 (GSM8K)WQ-AUC (MATH)BoN@32 (MATH)WQ-AUC (BBH)BoN@32 (BBH)
Llama-3.1-8BNormalized-step0.6785.30.7140.00.5868.0
Fixed-pos. pairwise0.7482.00.6942.70.7286.7
Stage-aligned †0.7388.00.8344.00.7489.3
PAIR0.7687.30.7746.70.7492.0
Qwen3-4BNormalized-step0.6880.70.6076.00.6294.6
Fixed-pos. pairwise0.8180.70.7779.30.6992.0
Stage-aligned †0.7884.00.8685.30.9096.0
PAIR0.9180.70.8984.70.8697.3
Gemma3-4BNormalized-step0.5788.00.5466.70.6296.0
Fixed-pos. pairwise0.5690.70.6173.30.6694.6
Stage-aligned †0.6690.00.6076.70.7897.3
PAIR0.6195.30.7475.30.7596.0
R1-Distill-8BNormalized-step0.5786.60.6680.00.84100.0
Fixed-pos. pairwise0.6588.70.7384.00.7794.6
Stage-aligned †0.7990.70.8986.70.88100.0
PAIR0.7589.30.8786.70.94100.0

† Stage-aligned uses step-level LLM annotation and serves as a reference. It is excluded from bold highlighting.

Normalized-step uses relative position without within-question comparison, and Fixed-pos. pairwise uses within-question comparison without relative position. PAIR combines both. It is never below Normalized-step (better in 21 of 24 cells, tied in 3) and outperforms Fixed-pos. pairwise in 23 of 24 cells (tied in 1). Each component therefore contributes, and their combination gives the strongest ranking signal. PAIR also performs close to Stage-aligned, which aligns steps by LLM-annotated functional stages, without needing any labels.

Normalizing by step count captures functional structure. Relative position can be measured in different ways, so we also checked which one best groups steps with similar roles. We labeled every reasoning step as Setup, Calculation, Verification, or Answer using Gemma-3-27B-IT with a fixed rubric (92% agreement with human labels on 200 sampled steps). For pairs of steps from two trajectories of the same question with different numbers of steps, we measure how often two steps in the same slot have the same stage. We compare phase alignment (normalized by step count) with Token-position normalization (normalized by token position) and absolute steps.

ModelDatasetPhase (PAIR)Absolute step (No normalization)Token-position normalizationChance
Llama-3.1-8BGSM8K0.7920.7400.7880.527
MATH-5000.6550.5920.6100.470
BBH0.6950.7260.7780.455
Qwen3-4BGSM8K0.6590.6960.6200.387
MATH-5000.5630.4740.4910.378
BBH0.6920.6780.6000.377
Gemma3-4BGSM8K0.8170.7480.7220.527
MATH-5000.6720.6440.5650.441
BBH0.7160.6630.6110.410
R1-Distill-8BGSM8K0.6610.6300.5430.434
MATH-5000.5740.5040.4810.412
BBH0.6350.5980.4870.347

Normalizing by step count matches the annotated stages better than normalizing by token position in 11 of 12 settings. Steps are the units in which the model organizes its reasoning, so aligning by step count groups steps that play similar roles more reliably. Results hold for K=3K = 3 to 66 (Appendix A.8).

Thank you for your detailed responses and clarifications. I appreciate the authors' efforts in addressing my concerns. I will maintain my positive rating.

Thank you for your positive assessment. We are glad our responses addressed your concerns.

Reviewer_C2LFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to better predict correctness from the hidden states of LLMs. The authors propose PAIR, which compares correct and incorrect reasoning trajectories from the same question at aligned phases. The learned directions are then used for trajectory ranking and activation steering.

Critical strengths
  1. The paper shows that the motivation, limitations of previous methods, and proposed solution are well explained. The overall presentation is easy to follow.

  2. The method shows promising results. The authors evaluate PAIR on multiple models and reasoning benchmarks, showing improvements in trajectory ranking and Best-of-N selection.

Critical weaknesses
  1. The alignments are based only on the specific thinking tokens, e.g., "then", "therefore", in a trajectory, without considering the actual content. For example, at P4, one trajectory may still be setting up equations, while another may be verifying its answer. I am not sure whether comparing these hidden states is meaningful enough. This paper mainly focuses on relatively simple reasoning benchmarks. This concern may become more serious for longer and more complex reasoning tasks, where trajectories involve more steps, backtracking, and corrections.

  2. The baselines are all traditional correctness probes. Therefore, it is unclear whether the improvements mainly come from comparing correct and incorrect trajectories of the same question. A simple baseline would be to extract hidden states at fixed relative positions (e.g., the second time some reasoning tokens ("then", "therefore",...) appear or the start of the second solving paragraph, etc.) from correct and incorrect trajectories of the same question, and train probes using the same pairwise ranking objective without phase alignment. This would help clarify whether other components, particularly phase alignment, provide additional benefits. Moreover, Table 3 shows that progressive steering provides little improvement over steering only at P1, with identical results on MATH and BBH. I think the authors should clarify why separate steps are necessary.

  3. The steps for using steering in inference are unclear. The method learns a separate steering direction for each phase (e.g., w1-w5 to P1–P5). However, during generation, the model does not know how many reasoning steps it will eventually produce. For example, if the model is currently at Step 3, this might correspond to P3 if the final trajectory contains five steps, but it could still be around P2 if the trajectory ends up being much longer. It is unclear how the authors determine which steering direction to apply during generation.

Questions for authors
  1. Since the paper evaluates trajectory ranking on four different LLMs, I would like to know whether the steering results also generalize to the other models.
6 comments
AuthorsOpenReview ↗

Response to Reviewer C2LF (1/3): W1

Thank you for your constructive feedback. We hope our responses will strengthen confidence in supporting our submission.

W1. Comparability of phase-aligned states and longer reasoning

Thank you for raising these concerns. Your comment raises two questions: whether hidden states at the same phase are comparable when phases do not look at content, and whether this holds for longer reasoning with backtracking. We address them in turn.

How often states in the same phase differ in content. To examine whether states at the same phase play comparable roles, as raised in your example, we labeled every reasoning step with its functional stage using Gemma-3-27B-IT and a fixed rubric: Setup (restating given information, defining variables, planning, or recalling a formula), Calculation (arithmetic, algebra, or logical deduction), Verification (checking or revising earlier work), and Answer (stating the final answer). On 200 randomly sampled steps, these labels agree with human labels 92% of the time. We then measured how often such mismatches occur and whether they affect PAIR.

For two trajectories of the same question with different numbers of steps, we take pairs of steps (s,s′)(s, s'), one from each trajectory. Let f(s)f(s) be the annotated stage of step ss, and let slot(s)\text{slot}(s) be the slot that an alignment method assigns it to. We measure how often two steps in the same slot have the same stage, P(f(s)=f(s′)∣slot(s)=slot(s′))P\big(f(s)=f(s')\mid \text{slot}(s)=\text{slot}(s')\big), for phase alignment (Eq. 6), absolute steps (min⁡(j,5)\min(j,5) for the jj-th step), and token fraction (the step's end position divided by trajectory length, in 5 equal bins). Chance is the agreement over all sampled step pairs, ignoring slots.

ModelDatasetPhase (PAIR)Absolute stepToken fractionChance
Llama-3.1-8BGSM8K0.7920.7400.7880.527
MATH-5000.6550.5920.6100.470
BBH0.6950.7260.7780.455
Qwen3-4BGSM8K0.6590.6960.6200.387
MATH-5000.5630.4740.4910.378
BBH0.6920.6780.6000.377
Gemma3-4BGSM8K0.8170.7480.7220.527
MATH-5000.6720.6440.5650.441
BBH0.7160.6630.6110.410
R1-Distill-8BGSM8K0.6610.6300.5430.434
MATH-5000.5740.5040.4810.412
BBH0.6350.5980.4870.347

Phase alignment gives the highest agreement in 10 of 12 settings and is well above chance in every setting. When PAIR compares two states at the same phase, they are more likely to be doing the same kind of work than under absolute steps or token position.

We also checked whether the remaining mismatches hurt PAIR. We retrained PAIR with slots defined directly by the annotated stages, which removes content mismatches by construction, and kept everything else the same (Table 2, "Stage-aligned").

ModelAlignmentWQ-AUC (GSM8K)BoN@32 (GSM8K)WQ-AUC (MATH)BoN@32 (MATH)WQ-AUC (BBH)BoN@32 (BBH)
Llama-3.1-8BStage (LLM)0.7388.00.8344.00.7489.3
Phase (PAIR)0.7687.30.7746.70.7492.0
Qwen3-4BStage (LLM)0.7884.00.8685.30.9096.0
Phase (PAIR)0.9180.70.8984.70.8697.3
Gemma3-4BStage (LLM)0.6690.00.6076.70.7897.3
Phase (PAIR)0.6195.30.7475.30.7596.0
R1-Distill-8BStage (LLM)0.7990.70.8986.70.88100.0
Phase (PAIR)0.7589.30.8786.70.94100.0

Content-based alignment performs close to phase alignment, so the mismatches that remain under phase alignment have little effect on the learned directions. Phase alignment thus offers a cheap substitute for content-based alignment, reaching similar performance without labeling any steps.

The results hold on long reasoning with backtracking. Our evaluation already includes DeepSeek-R1-Distill-Llama-8B, a reasoning model whose traces are long and contain frequent backtracking and re-checking (about 1,700 tokens on average on MATH-500). On this model, phase alignment matches the annotated stages better than both absolute steps and token fraction on all three benchmarks (table above). PAIR reaches WQ-AUC of 0.75, 0.87, and 0.94 on GSM8K, MATH-500, and BBH, and outperforms the same-objective baseline without phase alignment in all six cells of Table 2. These results are comparable to those on the other models, and the WQ-AUC on BBH (0.94) is the highest among all four models. Phase alignment therefore remains effective on long reasoning with frequent backtracking.

AuthorsOpenReview ↗

Response to Reviewer C2LF (2/3): W2

W2. Isolating Phase Alignment and the Role of Later Phases

Thank you for these suggestions. We added the baseline you suggested to Table 2 as "Fixed-pos. pairwise". It uses the same sampled trajectories, the same within-question pairs, and the same pairwise ranking objective as PAIR, but assigns steps by their absolute step index instead of their phase. We score it at the same slot as PAIR's final-phase scorer, so the only difference between the two is phase alignment.

ModelMethodWQ-AUC (GSM8K)BoN@32 (GSM8K)WQ-AUC (MATH)BoN@32 (MATH)WQ-AUC (BBH)BoN@32 (BBH)
Llama-3.1-8BFixed-pos. pairwise0.7482.00.6942.70.7286.7
PAIR0.7687.30.7746.70.7492.0
Qwen3-4BFixed-pos. pairwise0.8180.70.7779.30.6992.0
PAIR0.9180.70.8984.70.8697.3
Gemma3-4BFixed-pos. pairwise0.5690.70.6173.30.6694.6
PAIR0.6195.30.7475.30.7596.0
R1-Distill-8BFixed-pos. pairwise0.6588.70.7384.00.7794.6
PAIR0.7589.30.8786.70.94100.0

This added baseline separates the two components. Fixed-pos. pairwise already outperforms most of the pointwise correctness probes in Table 2, so comparing trajectories of the same question is an important source of the gain. PAIR still outperforms Fixed-pos. pairwise in 23 of 24 cells and ties in the remaining one, so phase alignment adds a consistent improvement on top of within-question training. We also report a controlled clustering comparison that separates the two factors in Appendix A.7.

Role of later phases. Separate phases matter because they serve different purposes. The final phase separates successful and unsuccessful trajectories best (Fig. 4a), and PAIR uses its direction for trajectory ranking and Best-of-N selection (Table 2). Steering, in contrast, works best at early phases. This holds on all four models in the revised Table 3 (see our response to Q1). So the phases that are easiest to read out are not the ones that are easiest to change. For steering itself, progressive steering is a safe default because it never falls below the baseline and does not require choosing a phase, whereas Single-P1 falls below the baseline on Gemma (GSM8K). We have revised Sec. 4.4 to make this distinction clearer and adjusted our claims about steering accordingly.

We think there are three reasons why later phases are easier to read out but harder to steer. (1) The model has usually settled on its answer by the later phases. Forced-answer accuracy goes up as the trajectory goes on (Fig. 3), consistent with prior findings that answers are often fixed before the reasoning is complete [1], so a late push has little to change. (2) The early steps are where the model decides how to solve the problem, while the later steps mostly carry out calculations and write down the answer. (3) The model generates one token at a time, and each token depends on all the tokens before it. A change at P1 affects everything the model writes after it, but by P4 or P5 most of the solution is already written and only a few steps are left. For the same reason, later hidden states have seen most of the solution, so they know more about whether the answer will be right, which makes them easier to read out.

[1] Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning.

AuthorsOpenReview ↗

Response to Reviewer C2LF (3/3): W3 and Q1

W3. Phase assignment during generation

Thank you for raising this point. Our steering experiments follow an offline protocol. We first generate an unsteered trajectory and segment it into steps, which determines the phase of each position. We then re-generate with the same decoding setup and add the learned direction of the corresponding phase at the same token positions. All directions are learned on training questions and applied to held-out test questions, so steering never uses the questions it is evaluated on. The goal of these experiments is to test whether the learned directions have a causal effect on reasoning, not to propose a ready-to-deploy steering method. We have revised Sec. 4.4 to state this clearly.

Q1. Steering on other models

Thank you for this question. We have extended the steering experiments to Qwen3-4B, Gemma3-4B, and DeepSeek-R1-Distill-Llama-8B, using the same protocol as for LLaMA (The full results for all four models are in Table 3 of the revised paper). For each model, directions are learned on training questions and applied to held-out test questions. To avoid confusion, we renamed the progressive schedule from "PAIR" to "Progressive" in Table 3, since all rows use directions learned by PAIR. Accuracy (%) is shown below:

ModelInterventionGSM8KMATHBBH
Llama-3.1-8BBaseline79.342.074.6
Single-P183.347.376.0
Progressive84.647.376.0
Qwen3-4BBaseline70.072.786.7
Single-P173.380.786.7
Progressive72.777.389.3
Gemma3-4BBaseline86.759.389.3
Single-P186.062.092.0
Progressive89.362.092.0
R1-Distill-8BBaseline86.075.380.0
Single-P187.378.782.7
Progressive86.775.385.3

The steering results generalize across models. Progressive steering improves accuracy over the unsteered baseline in 11 of 12 model–benchmark settings and never makes it worse. In every setting, the share of wrong answers flipped to correct is higher than the share of correct answers flipped to wrong (e.g., 30.0% vs. 1.5% on Gemma GSM8K), so the directions push generation toward correct answers rather than changing answers at random. Full results, including all single-phase interventions, flip rates, and output lengths, are in the revised Table 3.

Reviewer_pzMdOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address two limitations in existing works that probe hidden states of reasoning trajectories: (1) correctness probes trained across different questions might learn question difficulty instead of path quality; (2) aligning reasoning states by absolute step is problematic as they might correspond to different functional phases of reasoning. Their proposed method, Phase-Aligned Intra-question Reasoning (PAIR), samples multiple trajectories per question, maps each steps into relative phases, then learns a linear direction for each phase with a pairwise loss on successful vs. unsuccessful pairs. The learned directions are used for activation steering.

Critical strengths
  • The within-question analysis is interesting and the gap (Table 1) indeed shows the limitation of across-question correctness probing. I also liked the observation that early representations can predict final correctness even before the prefix is answerable (Figure 3), which implies that early-correctness signals might largely reflect question-level information and overstate path-quality. This justifies the within-question choice well.
Critical weaknesses
  • The proposed phase alignment design (Eq.6) is similar to the normalized step method: each trajectory is mapped into K relative phase slots, where K is the mode of the step counts in the training trajectories. This still did not address the absolute step problem in the motivating example (lines 73-75), phase 2 could mean setup in one solution and calculation in another. A more direct test would be to assign steps based on their functional stages such as setup, calculation, verification, and answers.

  • The effects of phase alignment is also not properly ablated. If I understand correctly, Fig 4b compares absolute steps across questions vs. phases within a single question, so two things change at once. The improved NMI might come from the within-question setting instead of phase alignment. I would want to see a controlled comparison on phase alignment as it is one of the two main claims in the paper.

  • Steering is evaluated only on Llama, and the progressive steering effects are very close to single phase P1 steering. On MATH and BBH, Single P1 and PAIR are identical. Does it mean subsequent phases are not useful?

Questions for authors
  • What if step markers are not present in the reasoning trajectories? How would you assign phases in this case? And how would you assign the phase during generation when the total step count is unknown?

  • Can the learned directions transfer across tasks or domains? If they need to be retrained on new tasks, this raises the question of generalizability of the paper.

9 comments
AuthorsOpenReview ↗

Response to Reviewer pzMd (1/3): W1

Thank you for your constructive feedback and we provide our answers below. We hope that our responses will strengthen confidence in supporting our submission.

W1. Phase Alignment vs. Functional Stages

We thank the reviewer for this insightful comment. Indeed, phase alignment maps steps by normalized progress rather than by semantic content, so a phase need not correspond to the same functional stage in every trajectory. It is designed as an inexpensive, annotation-free approximation of functional alignment. Our original wording overstated this and we have revised Sec. 3.2 and Sec. 5 accordingly.

Following your suggestion, we test this approximation directly. Using Gemma-3-27B-IT with a fixed rubric, we label every step as:

  • Setup: restating given information, defining variables, planning.
  • Calculation: arithmetic, algebra, or logical deduction.
  • Verification: checking earlier work.
  • Answer: stating the final answer.

This covers 4 models × 3 benchmarks (152k trajectories), and the labels agree with human labels on 92% of 200 sampled steps. Across all models, Setup concentrates in P1P_1 and Verification and Answer in the last phases (Appendix A.8).

Next, we use these labels to check whether steps that fall in the same phase actually do the same thing. We take pairs of steps (s,s′)(s, s'), where ss and s′s' come from two different trajectories of the same question that have different numbers of steps. Let f(s)f(s) be the annotated stage of step ss, and let slot(s)\text{slot}(s) be the slot that a slot assignment puts it in (for example, its phase under Eq. 6). We measure how often two steps in the same slot have the same stage:

Aslot=P(f(s)=f(s′)∣slot(s)=slot(s′)).A_{\text{slot}}=P\big(f(s)=f(s')\mid \text{slot}(s)=\text{slot}(s')\big).

We compare phase alignment, absolute steps, and token fraction (the step's end position divided by trajectory length, cut into 5 equal bins). Chance is the agreement between two randomly paired steps.

ModelDatasetPhase (PAIR)Absolute stepToken fractionChance
Llama-3.1-8BGSM8K0.7920.7400.7880.527
MATH-5000.6550.5920.6100.470
BBH0.6950.7260.7780.455
Qwen3-4BGSM8K0.6590.6960.6200.387
MATH-5000.5630.4740.4910.378
BBH0.6920.6780.6000.377
Gemma3-4BGSM8K0.8170.7480.7220.527
MATH-5000.6720.6440.5650.441
BBH0.7160.6630.6110.410
R1-Distill-8BGSM8K0.6610.6300.5430.434
MATH-5000.5740.5040.4810.412
BBH0.6350.5980.4870.347

Phase slots agree with the annotated stages more often than the other two slot assignments: phase alignment gives the highest agreement in 10 of 12 settings and beats token fraction in 11 of 12, so normalizing by step count captures functional stages better than normalizing by length. The gain over absolute steps is largest on MATH-500, where step counts vary most. The two exceptions have very short trajectories (about 3 steps), where Eq. 6 spreads steps over non-adjacent phases. Results hold for K=3K = 3 to 66 (Appendix A.8).

Aligning by annotated stage. We ran the test you suggested: we replace phase slots with stage slots and keep everything else fixed (Table 2, "Stage-aligned").

ModelAlignmentWQ-AUC (GSM8K)BoN@32 (GSM8K)WQ-AUC (MATH)BoN@32 (MATH)WQ-AUC (BBH)BoN@32 (BBH)
Llama-3.1-8BStage (LLM)0.7388.00.8344.00.7489.3
Phase (PAIR)0.7687.30.7746.70.7492.0
Qwen3-4BStage (LLM)0.7884.00.8685.30.9096.0
Phase (PAIR)0.9180.70.8984.70.8697.3
Gemma3-4BStage (LLM)0.6690.00.6076.70.7897.3
Phase (PAIR)0.6195.30.7475.30.7596.0
R1-Distill-8BStage (LLM)0.7990.70.8986.70.88100.0
Phase (PAIR)0.7589.30.8786.70.94100.0

Stage alignment requires LLM labels for every step, so we treat it as a reference point. PAIR performs close to it without any labels, and both clearly outperform alignment by absolute steps (Table 2). Normalizing by step count recovers most of what semantic labeling provides.

AuthorsOpenReview ↗

Response to Reviewer pzMd (2/3): W2 and W3

W2. Controlled Ablation of Phase Alignment

Thank you for pointing this out. To separate the two factors in Fig. 4b, we reran it as a controlled 2×2 comparison. In all four settings we cluster the same points (every step of 32 sampled trajectories per question) with the same features and the same procedure (z-score, PCA to 16 dimensions, K-means with K=5K=5). The only differences are whether we cluster all questions together or each question separately, and whether we score the clusters with absolute-step labels or phase labels (Eq. 6).

ModelAcross questionsWithin each question
Absolute stepPhaseAbsolute stepPhase
Llama-3.1-8B0.3190.3270.4780.492
Gemma3-4B0.2140.2160.5100.560
Qwen3-4B0.2060.2170.6080.652
R1-Distill-8B0.1870.1870.2840.283

Most of the NMI gain comes from comparing trajectories within the same question. Within the same question, phase labels still give higher NMI than absolute-step labels for Llama, Gemma, and Qwen, but not for R1-Distill. R1-Distill writes much longer solutions with a lot of backtracking and re-checking, so its hidden states do not cluster well by position under either labeling (NMI around 0.28, compared with 0.48 to 0.65 for the other models). We have added this comparison to Appendix A.7 and updated the description of Fig. 4b.

Clustering is only one way to look at this, so we also checked whether phase alignment helps trajectory ranking. We added a "Fixed-pos. pairwise" baseline to Table 2. It is trained exactly like PAIR, with the same within-question pairs, but it assigns steps by their absolute index instead of their phase. PAIR beats it in 23 of 24 cells, including all six cells for R1-Distill. So phase alignment itself improves ranking.

W3. Steering Beyond Llama and the Role of Later Phases

Other models and early-phase effects. We ran steering on Qwen3-4B, Gemma3-4B, and DeepSeek-R1-Distill-Llama-8B as well (revised Table 3). To avoid confusion, we renamed the progressive schedule from "PAIR" to "Progressive" in Table 3, since all rows use directions learned by PAIR. Accuracy (%) is shown below, and the full results, including all single-phase interventions, flip rates, and output lengths, are in Table 3 of the revised paper:

ModelInterventionGSM8KMATHBBH
Llama-3.1-8BBaseline79.342.074.6
Single-P183.347.376.0
Progressive84.647.376.0
Qwen3-4BBaseline70.072.786.7
Single-P173.380.786.7
Progressive72.777.389.3
Gemma3-4BBaseline86.759.389.3
Single-P186.062.092.0
Progressive89.362.092.0
R1-Distill-8BBaseline86.075.380.0
Single-P187.378.782.7
Progressive86.775.385.3

Across all four models, progressive steering improves accuracy over the unsteered baseline in 11 of 12 settings and never makes it worse. The new models also show the same pattern as LLaMA. Steering at P1 gives the largest and mostly positive gains, while steering only at P4 or P5 has small and inconsistent effects.

Role of later phases. Later phases are useful, but for a different purpose than early phases. The final phase separates successful and unsuccessful trajectories best (Fig. 4a), and PAIR uses its direction for trajectory ranking and Best-of-N selection (Table 2). Steering, in contrast, works best at early phases. So the phases that are easiest to read out are not the ones that are easiest to change. For steering itself, progressive steering is a safe default because it never falls below the baseline and does not require choosing a phase, whereas Single-P1 falls below the baseline on Gemma (GSM8K). We think there are three reasons why later phases are easier to read out but harder to steer. (1) The model has usually settled on its answer by the later phases: forced-answer accuracy goes up as the trajectory goes on (Fig. 3), consistent with prior findings that answers are often fixed before the reasoning is complete [1], so a late push has little to change. (2) The early steps are where the model decides how to solve the problem, while the later steps mostly carry out calculations and write down the answer. (3) The model generates one token at a time, and each token depends on all the tokens before it. A change at P1 affects everything the model writes after it, but by P4 or P5 most of the solution is already written and only a few steps are left. For the same reason, later hidden states have seen most of the solution, so they know more about whether the answer will be right, which makes them easier to read out. We have revised Sec. 4.4 to make this distinction clearer and adjusted our claims about steering accordingly.

[1] Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning.

AuthorsOpenReview ↗

Response to Reviewer pzMd (3/3): Q1 and Q2

Q1. Step segmentation and phase assignment during steering

Trajectories without explicit step markers. Our segmentation does not require explicit "Step k" markers. The marker rule (Appendix A.4.1) also accepts numbered lists, discourse transitions at line or sentence boundaries (First, Next, Then, Finally), heading lines, and the conclusion marker that anchors the final phase. When no marker is found, we fall back to splitting the trajectory at paragraph breaks (\n\n). The table below shows the fraction of trajectories each rule can segment:

ModelDatasetTrajectoriesStep markers onlyStep markers + fallback
Llama-3.1-8BGSM8K16,00098.7%99.99%
MATH13,28098.4%100%
BBH8,00098.9%100%
Qwen3-4BGSM8K16,00098.1%100%
MATH16,00097.6%100%
BBH8,00097.1%99.99%
Gemma3-4BGSM8K15,32499.5%100%
MATH11,66091.6%99.97%
BBH7,99499.4%100%
R1-Distill-8BGSM8K16,00099.5%99.99%
MATH16,00098.9%99.99%
BBH8,00097.4%100%
Total152,25898.0%99.99%

With the fallback, only 10 of the 152,258 trajectories cannot be segmented, and we drop these from training. We have added this coverage table to Appendix A.4.1.

Phase assignment during generation. Our steering experiments follow an offline protocol: phase boundaries are taken from the unsteered trajectory, and the learned directions are applied at the same token positions during re-generation. All directions are learned on training questions and applied to held-out test questions, so steering never uses the questions it is evaluated on. The goal of these experiments is to test whether the learned directions have a causal effect on reasoning, not to propose a deployable steering method. We have revised Sec. 4.4 to state this clearly.

Q2. Transfer across tasks

Thank you for this question. We would like to clarify one point: the same PAIR procedure applies to all four models and three benchmarks without task-specific changes. Whether a learned direction can be reused on a new task without retraining is a separate and important question, and we test it below.

We train PAIR on one benchmark and apply the learned final-phase scorer directly to the other two, without retraining or using any data from the target task. Each cell reports WQ-AUC / BoN@32. Diagonal entries (in bold) are in-task results from Table 2. We have added these results to Appendix A.9.

ModelTrain ↓ / Test →GSM8KMATH-500BBH
Llama-3.1-8BGSM8K0.763 / 87.30.721 / 49.30.714 / 66.7
MATH-5000.741 / 87.30.767 / 46.70.703 / 76.0
BBH0.721 / 86.70.707 / 40.00.738 / 92.0
Qwen3-4BGSM8K0.905 / 80.70.802 / 82.00.822 / 97.3
MATH-5000.652 / 74.70.892 / 84.70.521 / 68.0
BBH0.801 / 87.30.803 / 82.00.856 / 97.3
Gemma3-4BGSM8K0.606 / 95.30.529 / 66.00.640 / 94.7
MATH-5000.584 / 90.70.740 / 75.30.613 / 97.3
BBH0.593 / 86.70.573 / 64.00.754 / 96.0
R1-Distill-8BGSM8K0.751 / 89.30.731 / 82.70.635 / 72.0
MATH-5000.654 / 92.70.871 / 86.70.832 / 96.0
BBH0.598 / 86.70.684 / 86.00.942 / 100.0

The learned directions transfer well across tasks without any retraining. Every transferred direction stays above chance, and averaged over models, WQ-AUC is 0.68 when the direction comes from a different benchmark, compared with 0.80 in-task. On LLaMA, the drop is at most 0.06 in every transfer pair. This suggests that the path-quality directions capture a signal that is largely shared across math and logical reasoning tasks. BoN@32 is noisier because it depends only on the top-ranked trajectory, and in several cases the transferred direction even matches or exceeds the in-task result.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Reviewers and Area Chair,

Thank you for your time and efforts in evaluating our paper. We are encouraged to see the reviewers' recognition of our paper's strengths, including the identification of an important limitation in across-question correctness probing (ksno, pzMd), the well-justified within-question design (pzMd), a well-motivated method (ksno, C2LF), consistent gains in trajectory ranking and Best-of-N selection across models and benchmarks (ksno, C2LF), and clear presentation (C2LF). We summarize below the new results and revisions, and we answer every point of each review in the individual responses.

Key Contributions

  • We show that across-question correctness probes partly rely on question-level information rather than path quality, and that absolute step indices can misalign reasoning stages.
  • PAIR compares successful and unsuccessful trajectories of the same question at matched relative phases and learns phase-specific path-quality directions without step-level annotation.
  • Across four models and three reasoning benchmarks, PAIR improves trajectory ranking and Best-of-N selection, and phase-wise steering provides causal evidence for the learned directions.

Key Revisions

The revised PDF marks all changes in blue.

  • Functional stages (pzMd, C2LF). We labeled every step as Setup, Calculation, Verification, or Answer using Gemma-3-27B-IT. Steps in the same phase share the same stage more often than under absolute steps or token position. We also trained a Stage-aligned variant that aligns steps by these labels. PAIR performs close to it without using any labels (Table 2, App. A.8).
  • Isolating phase alignment (pzMd, C2LF, ksno). A new Fixed-pos. pairwise baseline uses the same within-question pairs and objective as PAIR but aligns steps by absolute index. PAIR outperforms it in 23 of 24 cells and is never below Normalized-step. A controlled 2×2 clustering comparison further separates the two factors (Table 2, App. A.7).
  • Steering on all models (pzMd, C2LF). Progressive steering improves accuracy on all four models in 11 of 12 settings and never lowers it. Early phases are most effective for intervention, while the final phase is most informative for readout (Table 3, Sec. 4.4).
  • Practical use (C2LF). PAIR scores sampled trajectories from their hidden states, fits naturally into test-time scaling, and requires only a single generation pass. The offline protocol applies only to the steering experiments, which serve as a causal test (Sec. 4.4).
  • Generality (pzMd). Step segmentation does not require explicit step markers and covers 99.99% of 152,258 trajectories. The learned directions transfer across benchmarks without retraining (App. A.4.1, A.9).
  • Clarity (ksno). We revised the abstract, introduction, and Sec. 3 and 4 to state the motivation and method more directly.
  • Significance and novelty (pzMd). Pooled over all settings, PAIR outperforms each baseline in nearly all cells of Table 2, and progressive steering corrects significantly more answers than it breaks (exact McNemar test, p < 0.001), App. A.10. The contribution list in the introduction now states how the components of PAIR address the diagnosed issues.

After the rebuttal, all three reviewers acknowledged our responses. Reviewers ksno and C2LF maintained their positive scores, and Reviewer pzMd raised the score to positive.

We hope these results and revisions further strengthen your confidence in our work, and we would be happy to provide any further clarification. Thank you again for your time and consideration.

Sincerely,

Authors of Submission 22404

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.