- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Hidden-Align, a lightweight auxiliary loss that improves LLM reasoning by aligning the hidden states of correct reasoning rollouts during RL. The key observation is that different correct rollouts naturally exhibit similar hidden representations before generating the final answer, despite following different reasoning paths. By maximizing their cosine similarity at this position, Hidden-Align encourages the model to learn a more consistent representation of correct decisions. Experiments across multiple model scales and reasoning benchmarks show improvements over standard RLVR methods, with negligible training overhead and no additional inference cost.
Critical strengths
-
The paper presents a clear and well-motivated approach, with a natural progression from empirical observations to the proposed method. The analysis of hidden-state similarities provides intuitive justification for the design choices of Hidden-Align.
-
Hidden-Align consistently improves performance across multiple math reasoning benchmarks and model scales. It also shows generalizability across different RL algorithms, model families, and tasks, including spatial reasoning.
Critical weaknesses
-
The current analyses mainly show increased hidden-state similarity and improved accuracy, without directly examining whether the learned representations better capture correctness-related information or whether reasoning-path diversity is preserved. More analyses, such as correctness probing or reasoning diversity measurements, would help substantiate the proposed explanation.
-
The performance gains of Hidden-Align appear a bit inconsistent across different settings. For example, the improvement on Qwen3-4B varies from 1.38 to 6.21 points across different training seeds. Also, changing the answer format to a natural-language marker reduces the average gain to only 0.53 percentage points. These variations suggest that the effectiveness of Hidden-Align may be sensitive to training randomness and output formats.
Questions for authors
-
Does aligning the hidden states of correct rollouts affect the diversity of their reasoning paths?
-
LLM reasoning can exhibit misalignment between the reasoning process and the final answer, so a correct answer does not necessarily imply a valid reasoning trajectory. Since Hidden-Align treats all rollouts with correct final answers as equally valid alignment targets, have the authors investigated whether the quality of these reasoning trajectories affects the effectiveness of the proposed method, particularly when some rollouts arrive at correct answers through flawed reasoning?
4 comments
Response to Reviewer maGY (2/3)
W2: Variation across training seeds and output formats
The performance gains of Hidden-Align appear a bit inconsistent across different settings. For example, the improvement on Qwen3-4B varies from 1.38 to 6.21 points across different training seeds. Also, changing the answer format to a natural-language marker reduces the average gain to only 0.53 percentage points. These variations suggest that the effectiveness of Hidden-Align may be sensitive to training randomness and output formats.
Hidden-Align improves the eight-task average in all three training seeds. DAPO scores 51.97, 53.46, and 51.56, while Hidden-Align scores 58.18, 54.84, and 54.37. The across-seed mean increases from 52.33 ± 1.00 to 55.80 ± 2.08 (+3.47 points). The magnitude varies, but the improvement is positive in all three independent runs.
Different seeds generate different training rollouts, and Hidden-Align draws supervision from correct pairs within those rollouts. The number of such pairs, the problems they cover, and their reasoning content affect the alignment signal received during training, providing an explanation for variation in gain magnitude.
We also repeated two anchor and two weight ablations with the additional seed. The default remains the highest-scoring configuration in this comparison, leading by 1.33–3.07 points on the eight-task average and providing further support for the choices in the paper:
| Configuration | Seed A: four-task avg. | Seed B: four-task avg. | Seed B: eight-task avg. | Default advantage (Seed B, eight tasks, pp) |
|---|---|---|---|---|
| Default: pre_boxed, λ=0.001 | 49.60 | 46.55 | 55.75 | — |
| Answer position (mean over answer-token states) | 42.96 | 44.86 | 54.42 | +1.33 |
| pre_boxed_open | 43.92 | 43.94 | 53.82 | +1.93 |
| λ=0.0005 | 43.76 | 43.33 | 52.74 | +3.01 |
| λ=0.0001 | 42.91 | 42.66 | 52.68 | +3.07 |
The four-task average follows Tables 5 and 13: AIME24, AIME26, AMC24, and HMMT, allowing a cross-seed comparison with the original ablations. It differs from the average excluding validation-mixture benchmarks, which replaces AIME24 with OlympiadBench.
For answer is, our hypothesis is that the pre-marker representation encodes the preceding reasoning while drawing on semantic, syntactic, and collocational knowledge acquired during pretraining to predict the introductory phrase. Alignment may therefore act more on language production and contribute less directly to the answer decision. The matched comparison still yields a +0.53-point average gain under this convention, showing transfer to another answer format. The gain magnitude can depend on how much answer-decision information the chosen boundary carries.
Response to Reviewer maGY (3/3)
Q2: Correct answers with flawed reasoning
LLM reasoning can exhibit misalignment between the reasoning process and the final answer, so a correct answer does not necessarily imply a valid reasoning trajectory. Since Hidden-Align treats all rollouts with correct final answers as equally valid alignment targets, have the authors investigated whether the quality of these reasoning trajectories affects the effectiveness of the proposed method, particularly when some rollouts arrive at correct answers through flawed reasoning?
We expanded the process audit to address this question. Using existing evaluation outputs, from each of four mathematical benchmarks, we sampled six problems for which both methods produced correct answers, then selected one answer-correct response per method per problem: 24 problems and 48 responses in total. Sampling followed a fixed rule. We hid method identities during review, checked each retained final derivation, and independently verified key calculations:
| Method | Supported derivation | Substantive derivation error | Key proof gap | Total |
|---|---|---|---|---|
| DAPO | 20 | 0 | 4 | 24 |
| Hidden-Align | 21 | 2 | 1 | 24 |
This was a model-assisted process audit, with corrected exploratory errors recorded separately from errors retained in the final derivation. Of the 48 responses, 41 had supported derivations, two contained substantive errors, and five had key proof gaps. For example, one response treated a required maximization as minimization and then reversed a candidate comparison; the two errors canceled to produce the correct answer. This shows that even an output with a correct, verified final answer can contain reasoning errors or key proof gaps.
Regarding their effect on training, a flawed trajectory can introduce noise into alignment, but each trajectory’s reference direction is formed jointly by other successful trajectories for the same problem. When a group contains several reliable trajectories and their errors do not reinforce one another, the shared reference can dilute an isolated error’s contribution, whereas an error repeated across trajectories may persist. Training remains driven by the RL reward, with a small alignment weight adding a constraint on answer-boundary representations; erroneous steps in an individual trace are not treated as text to imitate word for word. We view this joint reference as a mechanistic hypothesis for learning from noisy answer-level supervision. The paper’s gains across scales and seeds were obtained under actual answer-level verification; the added audit characterizes process quality in the sampled evaluation outputs.
Thank you for the detailed response. Most of my concerns have been addressed. I will maintain my positive score.
Response to Reviewer maGY (1/3)
Thank you for the questions about representations, diversity, stability, and the quality of successful reasoning trajectories.
W1 / Q1: Decision representations and reasoning diversity
The added text analysis shows no decrease in overall textual diversity among answer-correct outputs. We analyzed 17,280 existing responses across two training seeds, four mathematical benchmarks, and 135 distinct questions. We retained questions with at least two correct responses from each method, yielding 83 eligible questions per seed. We matched the number of correct rollouts per question, computed word/punctuation 5-gram Jaccard distance over the full reasoning text, and averaged the four benchmark scores equally. Higher distance means greater textual variation.
Both changes are small on the metric’s 0–1 scale and positive in direction. Local analysis also finds more similar opening language but greater variation in the middle of seed B traces. Answer-boundary alignment can therefore accompany local changes in wording while retaining substantial variation across full reasoning texts.
For decision representations, Appendix A, Figure 7 compares all correct and all incorrect rollouts in aggregate, without further grouping incorrect rollouts by answer identity. The base Qwen3-4B model generates 10 rollouts for each of 1,000 problems, yielding three types of within-problem pairs. The anchor similarities pooled over eligible pairs are summarized below; the figure provides the full distributions.
Correct pairs are more similar in the aggregate comparison. The added same-wrong-answer control shows that a shared wrong answer can also correspond to similar representations: across 211 eligible problems, the question-averaged similarities of correct pairs and same-wrong-answer pairs are 0.836 and 0.823, with a 95% CI of [−0.028, 0.055] for the difference. Our interpretation is that shared answer information can be part of a shared decision representation, while the verifier determines which trajectories to align. Hidden-Align encourages successful trajectories to form representations supporting the correct answer at the answer boundary, without requiring token-by-token matching of intermediate reasoning text. The text analysis above further shows that answer-boundary alignment can coexist with varied reasoning expression.