acceptodds
Under review as a conference paper at ICLR 2027

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ≈ 0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified “correct decision” representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, without appreciable training overhead or additional inference cost. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales. These benefits generalize across RL algorithms and model families, extending beyond mathematics to spatial reasoning. Systematic ablations support the design choices.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, mathematical reasoning, hidden state alignment, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_maGYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Hidden-Align, a lightweight auxiliary loss that improves LLM reasoning by aligning the hidden states of correct reasoning rollouts during RL. The key observation is that different correct rollouts naturally exhibit similar hidden representations before generating the final answer, despite following different reasoning paths. By maximizing their cosine similarity at this position, Hidden-Align encourages the model to learn a more consistent representation of correct decisions. Experiments across multiple model scales and reasoning benchmarks show improvements over standard RLVR methods, with negligible training overhead and no additional inference cost.

Critical strengths
  1. The paper presents a clear and well-motivated approach, with a natural progression from empirical observations to the proposed method. The analysis of hidden-state similarities provides intuitive justification for the design choices of Hidden-Align.

  2. Hidden-Align consistently improves performance across multiple math reasoning benchmarks and model scales. It also shows generalizability across different RL algorithms, model families, and tasks, including spatial reasoning.

Critical weaknesses
  1. The current analyses mainly show increased hidden-state similarity and improved accuracy, without directly examining whether the learned representations better capture correctness-related information or whether reasoning-path diversity is preserved. More analyses, such as correctness probing or reasoning diversity measurements, would help substantiate the proposed explanation.

  2. The performance gains of Hidden-Align appear a bit inconsistent across different settings. For example, the improvement on Qwen3-4B varies from 1.38 to 6.21 points across different training seeds. Also, changing the answer format to a natural-language marker reduces the average gain to only 0.53 percentage points. These variations suggest that the effectiveness of Hidden-Align may be sensitive to training randomness and output formats.

Questions for authors
  1. Does aligning the hidden states of correct rollouts affect the diversity of their reasoning paths?

  2. LLM reasoning can exhibit misalignment between the reasoning process and the final answer, so a correct answer does not necessarily imply a valid reasoning trajectory. Since Hidden-Align treats all rollouts with correct final answers as equally valid alignment targets, have the authors investigated whether the quality of these reasoning trajectories affects the effectiveness of the proposed method, particularly when some rollouts arrive at correct answers through flawed reasoning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer maGY (1/3)

Thank you for the questions about representations, diversity, stability, and the quality of successful reasoning trajectories.

W1 / Q1: Decision representations and reasoning diversity

Does aligning the hidden states of correct rollouts affect the diversity of their reasoning paths?

The added text analysis shows no decrease in overall textual diversity among answer-correct outputs. We analyzed 17,280 existing responses across two training seeds, four mathematical benchmarks, and 135 distinct questions. We retained questions with at least two correct responses from each method, yielding 83 eligible questions per seed. We matched the number of correct rollouts per question, computed word/punctuation 5-gram Jaccard distance over the full reasoning text, and averaged the four benchmark scores equally. Higher distance means greater textual variation.

Training seedFull-trace distance change (HA−DAPO)95% CI
A+0.0363[0.0173, 0.0567]
B+0.0084[−0.0122, 0.0296]

Both changes are small on the metric’s 0–1 scale and positive in direction. Local analysis also finds more similar opening language but greater variation in the middle of seed B traces. Answer-boundary alignment can therefore accompany local changes in wording while retaining substantial variation across full reasoning texts.

For decision representations, Appendix A, Figure 7 compares all correct and all incorrect rollouts in aggregate, without further grouping incorrect rollouts by answer identity. The base Qwen3-4B model generates 10 rollouts for each of 1,000 problems, yielding three types of within-problem pairs. The anchor similarities pooled over eligible pairs are summarized below; the figure provides the full distributions.

Pair typeMean cosine similarity pooled over pairs
Correct–correct0.854
Correct–wrong0.744
Wrong–wrong0.808

Correct pairs are more similar in the aggregate comparison. The added same-wrong-answer control shows that a shared wrong answer can also correspond to similar representations: across 211 eligible problems, the question-averaged similarities of correct pairs and same-wrong-answer pairs are 0.836 and 0.823, with a 95% CI of [−0.028, 0.055] for the difference. Our interpretation is that shared answer information can be part of a shared decision representation, while the verifier determines which trajectories to align. Hidden-Align encourages successful trajectories to form representations supporting the correct answer at the answer boundary, without requiring token-by-token matching of intermediate reasoning text. The text analysis above further shows that answer-boundary alignment can coexist with varied reasoning expression.

AuthorsOpenReview ↗

Response to Reviewer maGY (2/3)

W2: Variation across training seeds and output formats

The performance gains of Hidden-Align appear a bit inconsistent across different settings. For example, the improvement on Qwen3-4B varies from 1.38 to 6.21 points across different training seeds. Also, changing the answer format to a natural-language marker reduces the average gain to only 0.53 percentage points. These variations suggest that the effectiveness of Hidden-Align may be sensitive to training randomness and output formats.

Hidden-Align improves the eight-task average in all three training seeds. DAPO scores 51.97, 53.46, and 51.56, while Hidden-Align scores 58.18, 54.84, and 54.37. The across-seed mean increases from 52.33 ± 1.00 to 55.80 ± 2.08 (+3.47 points). The magnitude varies, but the improvement is positive in all three independent runs.

Different seeds generate different training rollouts, and Hidden-Align draws supervision from correct pairs within those rollouts. The number of such pairs, the problems they cover, and their reasoning content affect the alignment signal received during training, providing an explanation for variation in gain magnitude.

We also repeated two anchor and two weight ablations with the additional seed. The default remains the highest-scoring configuration in this comparison, leading by 1.33–3.07 points on the eight-task average and providing further support for the choices in the paper:

ConfigurationSeed A: four-task avg.Seed B: four-task avg.Seed B: eight-task avg.Default advantage (Seed B, eight tasks, pp)
Default: pre_boxed, λ=0.00149.6046.5555.75—
Answer position (mean over answer-token states)42.9644.8654.42+1.33
pre_boxed_open43.9243.9453.82+1.93
λ=0.000543.7643.3352.74+3.01
λ=0.000142.9142.6652.68+3.07

The four-task average follows Tables 5 and 13: AIME24, AIME26, AMC24, and HMMT, allowing a cross-seed comparison with the original ablations. It differs from the average excluding validation-mixture benchmarks, which replaces AIME24 with OlympiadBench.

For answer is, our hypothesis is that the pre-marker representation encodes the preceding reasoning while drawing on semantic, syntactic, and collocational knowledge acquired during pretraining to predict the introductory phrase. Alignment may therefore act more on language production and contribute less directly to the answer decision. The matched comparison still yields a +0.53-point average gain under this convention, showing transfer to another answer format. The gain magnitude can depend on how much answer-decision information the chosen boundary carries.

AuthorsOpenReview ↗

Response to Reviewer maGY (3/3)

Q2: Correct answers with flawed reasoning

LLM reasoning can exhibit misalignment between the reasoning process and the final answer, so a correct answer does not necessarily imply a valid reasoning trajectory. Since Hidden-Align treats all rollouts with correct final answers as equally valid alignment targets, have the authors investigated whether the quality of these reasoning trajectories affects the effectiveness of the proposed method, particularly when some rollouts arrive at correct answers through flawed reasoning?

We expanded the process audit to address this question. Using existing evaluation outputs, from each of four mathematical benchmarks, we sampled six problems for which both methods produced correct answers, then selected one answer-correct response per method per problem: 24 problems and 48 responses in total. Sampling followed a fixed rule. We hid method identities during review, checked each retained final derivation, and independently verified key calculations:

MethodSupported derivationSubstantive derivation errorKey proof gapTotal
DAPO200424
Hidden-Align212124

This was a model-assisted process audit, with corrected exploratory errors recorded separately from errors retained in the final derivation. Of the 48 responses, 41 had supported derivations, two contained substantive errors, and five had key proof gaps. For example, one response treated a required maximization as minimization and then reversed a candidate comparison; the two errors canceled to produce the correct answer. This shows that even an output with a correct, verified final answer can contain reasoning errors or key proof gaps.

Regarding their effect on training, a flawed trajectory can introduce noise into alignment, but each trajectory’s reference direction is formed jointly by other successful trajectories for the same problem. When a group contains several reliable trajectories and their errors do not reinforce one another, the shared reference can dilute an isolated error’s contribution, whereas an error repeated across trajectories may persist. Training remains driven by the RL reward, with a small alignment weight adding a constraint on answer-boundary representations; erroneous steps in an individual trace are not treated as text to imitate word for word. We view this joint reference as a mechanistic hypothesis for learning from noisy answer-level supervision. The paper’s gains across scales and seeds were obtained under actual answer-level verification; the added audit characterizes process quality in the sampled evaluation outputs.

Thank you for the detailed response. Most of my concerns have been addressed. I will maintain my positive score.

Reviewer_TxPmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hidden-Align, an auxiliary representation-alignment loss for reinforcement learning with verifiable rewards (RLVR). The method aligns the last-layer hidden states of multiple verified-correct rollouts for the same prompt at an anchor token immediately preceding the answer marker. The auxiliary cosine-similarity loss is combined with the standard DAPO policy optimization objective, with the goal of encouraging correct reasoning trajectories to share a more consistent representation near the answer boundary. They evaluate Hidden-Align on eight mathematical reasoning benchmarks using Qwen3 models at 1.7B, 4B, and 14B scales. They report average pass@1 improvements over DAPO of 3.8, 6.2, and 5.4 percentage points, respectively. The paper also reports improvements in pass@k, results with GSPO and Llama-3.2-3B-Instruct, and an extension to spatial reasoning using a vision-language model. Ablation studies investigate the choice of alignment objective, token position, transformer layer, and loss weight.

Critical strengths

1.Hidden-Align targets a specific point in the generation process: the last-layer hidden state immediately before the answer marker. The method is conceptually straightforward and integrates an auxiliary cosine-similarity loss into the existing RL objective. It does not require an additional teacher model or an inference-time modification, making the approach potentially useful for existing RLVR pipelines.

2.Promising improvements across model scales and mathematical benchmarks. The main results show average accuracy gains of 3.8, 6.2, and 5.4 percentage points over DAPO for Qwen3-1.7B, Qwen3-4B, and Qwen3-14B, respectively. The gains are substantial on several challenging competition-math benchmarks, including AIME and HMMT. The reported pass@k curves also suggest that the benefits are not limited to single-sample accuracy.

3.The paper compares alignment against several alternative objectives, including separation, diversity, negative-sample repulsion, and global clustering. It also evaluates different token positions and transformer layers. The reported results favor aligning correct rollouts at the selected answer-boundary position using the last transformer layer. These comparisons provide meaningful evidence that the proposed design choices matter.

  1. The paper reports positive average gains when combining Hidden-Align with GSPO, when training a Llama model, and on a spatial-reasoning evaluation using a vision-language model. Although the magnitude and consistency of gains vary across settings, these results suggest that the method may have broader applicability than a single Qwen3 mathematical-reasoning configuration.
Critical weaknesses

1.The paper claims that Hidden-Align introduces no appreciable training overhead. However, the main body does not provide enough detail to independently assess the incremental computational cost of obtaining and aligning the reference representations. Since training efficiency is an important part of the method's practical appeal, a clearer accounting of additional forward computation, memory use, and end-to-end training time would strengthen this claim.

2.Although the paper reports improvements across several model families and task types, some individual benchmark scores decrease. For example, the Llama experiment shows lower accuracy on AMC24 and Minerva Math, while the spatial-reasoning results include a slight decrease on EmbSpatialBench. The paper would benefit from a more explicit discussion of when the method is most effective and which task characteristics may limit its benefits.

3.The primary formulation assumes that a suitable answer marker can be identified and that correctness can be verified. This is reasonable for the main mathematical-reasoning setting, but identifying a semantically appropriate anchor may be less straightforward for tasks with different output formats or less clearly defined answer boundaries. The method's broader applicability therefore depends on additional task-specific design choices.

Questions for authors

1.Could the authors clarify whether the reference-representation collection reuses a forward pass already required by the baseline or requires an additional pass? Under the main experimental configuration, could the authors report the incremental forward-pass computation, peak memory usage, and end-to-end training time relative to DAPO under matched training settings?

2.Based on the reported generalization experiments, what factors might explain the variation in gains across benchmarks, including the decreases on some individual benchmarks? When adapting Hidden-Align to a new task or output format, how is a suitable anchor position selected and validated?

3 comments
AuthorsOpenReview ↗

Response to Reviewer TxPm (1/2)

Thank you for the questions about training efficiency and the conditions under which Hidden-Align is effective.

W1 / Q1: Forward computation, memory, and training time

Could the authors clarify whether the reference-representation collection reuses a forward pass already required by the baseline or requires an additional pass? Under the main experimental configuration, could the authors report the incremental forward-pass computation, peak memory usage, and end-to-end training time relative to DAPO under matched training settings?

The current implementation reuses forward computation already required by DAPO. It collects detached references while computing old-policy log probabilities, obtains the representations to align during the policy-update forward, and backpropagates the alignment and RL losses together. No additional full-model forward is required.

We added matched profiling with hidden-state monitoring disabled for DAPO. On eight H20D GPUs, the averages over 13 common recorded steps after excluding the initial cold start are:

MetricDAPOHidden-Align
Old-policy forward, seconds/step5.238.53
Actor update, seconds/step18.5719.10
Training step excluding validation, seconds198.89198.74
Monitored per-GPU peak memory range, GiB88.14–95.0188.14

The alignment-related stages add 3.82 seconds per step, approximately 1.9% of the baseline training-step time, with no higher memory peak observed in device monitoring. Whole-step time also varies with rollout length; both methods took approximately 199 seconds per step in this measurement.

For the main experimental scale, Appendix B, Table 7 reports end-to-end estimates for 200 steps on four H200 GPUs: 53.61 ± 2.37 hours for DAPO and 52.93 ± 3.20 hours for Hidden-Align, reconstructed from checkpoint timestamps (mean ± sample SD over three runs per method). The short profile quantifies the current implementation’s added computation, while the full-run records show comparable total training cost in the main experiments.

AuthorsOpenReview ↗

Response to Reviewer TxPm (2/2)

W2 / Q2 (benchmark differences): Conditions affecting gains

Based on the reported generalization experiments, what factors might explain the variation in gains across benchmarks, including the decreases on some individual benchmarks?

We expect Hidden-Align to help most when a model can already generate several correct solutions to a problem, but their answer-boundary representations remain inconsistent. Successful trajectories can then provide additional constraints for one another. If correct solutions are rare, few prompt groups supply alignment pairs. If the main bottleneck is visual recognition, information acquisition, or missing knowledge, an answer-boundary constraint has less direct access to that bottleneck.

This offers an explanation for variation in the transfer results. Benchmarks place different demands on knowledge, perception, and multi-step reasoning, so the same alignment weight may benefit these abilities unevenly. For the smaller Llama model, the additional constraint may also redistribute limited representational capacity: greater consistency on some tasks can come at the expense of fine distinctions useful on others. In spatial tasks dominated by perceptual errors, aligning the end of reasoning may likewise provide a smaller benefit. We identify correct-trajectory availability, the task’s main bottleneck, and the balance between alignment and the RL objective as three factors affecting transfer. These are explanations motivated by the objective; the weight ablation, where weak alignment yields smaller gains and overly strong alignment causes divergence, further motivates matching the constraint strength to the model and task.

W3 / Q2 (anchor selection): Adapting and validating the anchor

When adapting Hidden-Align to a new task or output format, how is a suitable anchor position selected and validated?

For transfer, we use the output structure to locate the transition from reasoning to answer production. The mathematical experiments use the token before the answer marker, and spatial reasoning uses the token before </think>, with the loss unchanged. The natural-language answer-is experiment further shows that the anchor can adapt to the answer convention.

For a new task, first fix the correctness verifier and output convention, then check that the candidate boundary can be extracted consistently. If several nearby positions are plausible, compare a small candidate set on a separate development set with the remaining training settings held fixed, then fix the position before final evaluation. The paper’s position ablation supports this procedure: reasoning tokens, the pre-answer boundary, and positions within the answer produce different gains. The transferable principle is the answer boundary; the task format determines the specific token.

Thank you for taking the time to write such a detailed response. Your clarifications and additional results have addressed my concerns. I will maintain my current score.

Reviewer_NUQ3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Hidden-Align, an auxiliary loss for RLVR that aligns the last-layer hidden states of correct rollouts for the same prompt at the token preceding the answer marker. Experiments on eight mathematical reasoning benchmarks report improvements over DAPO across three Qwen3 model sizes, with further evaluations on GSPO, Llama, and spatial reasoning.

Critical strengths
  1. Interesting idea with a simple formulation. The paper exploits agreement among verified correct rollouts as an additional training signal beyond binary rewards. Aligning their hidden states at the answer boundary is an interesting and easy-to-understand approach that complements the existing RL objective.
  2. Empirical support across multiple settings. The method improves performance across three Qwen3 model sizes, with positive average gains in all three Qwen3-4B training seeds. Further evaluations with GSPO, Llama, and spatial reasoning broaden the evidence for its applicability.
  3. Thorough ablations and useful controls. The paper systematically examines the loss type, anchor position, layer, and alignment weight. Additional comparisons with logit alignment and analyses under matched answer formats help assess alternative explanations for the improvements.
Critical weaknesses
  1. The mechanistic interpretation is not fully established. The results support the usefulness of alignment, but do not directly establish that it extracts a shared “correct-decision” representation. Higher similarity among correct rollouts could partly reflect their shared answer identity. Comparing against incorrect rollouts that produce the same wrong answer would help distinguish these explanations.
  2. Applicability across answer conventions remains uncertain. Although the spatial reasoning experiments provide evidence beyond mathematics, the method requires task-specific anchor selection. The alternative answer is setting yields only a 0.53-point average gain. Since this experiment changes the marker, verifier, and anchor rule together, it remains unclear which factors determine successful transfer.
  3. Checkpoint-selection independence needs clarification. The supplementary validation construction includes benchmarks also used for final evaluation, while Appendix B states that the 1.7B checkpoints were selected by validation performance. The authors should clarify whether this overlapping mixture informed checkpoint selection and distinguish selection benchmarks from independently held-out evaluations. This is a model-selection concern, not evidence that test examples were used for gradient updates.
Questions for authors
  1. How should the anchor token be selected for a new task or answer format?

  2. Have you compared the hidden-state similarity of correct–correct pairs with incorrect pairs sharing the same wrong answer?

  3. Could you clarify how checkpoints were selected and whether the validation data overlap with the final evaluation benchmarks?

  4. Are the preferred anchor position and alignment weight consistent across training seeds?

  5. What might explain the smaller improvement with the natural-language answer is marker?

  6. Could you clarify the additional training cost relative to a DAPO baseline with hidden-state monitoring disabled?

3 comments
AuthorsOpenReview ↗

Response to Reviewer NUQ3 (1/3)

Thank you for the questions about the alignment mechanism, transfer, evaluation, and computational cost.

W2 (anchor selection) / Q1: Selecting an anchor for a new task

How should the anchor token be selected for a new task or answer format?

Our anchor-selection principle is to use the representation at the transition from reasoning to final-answer production. The mathematical experiments use the token before the answer marker; spatial reasoning uses the token before </think>. Both apply the same alignment objective. The spatial and answer-is experiments show how this principle transfers across task structures and answer conventions.

In practice, identify the boundary from the output format and check that it can be extracted consistently. If several candidates are plausible, compare a small set on a separate development set, then fix the choice before final evaluation.

W1 / Q2: Correct-decision representations and same-wrong-answer controls

Have you compared the hidden-state similarity of correct–correct pairs with incorrect pairs sharing the same wrong answer?

Following the suggestion, we compared correct–correct pairs with pairs producing the same wrong answer within each problem. A problem was included if it had at least two correct rollouts and at least two incorrect rollouts sharing an answer. We normalized equivalent numerical expressions and required the answer associated with the cached anchor to agree with the final answer. In the base-model cache, 211 problems met these conditions. The question-averaged cosine similarities were 0.836 and 0.823, with a 95% CI of [−0.028, 0.055] for their difference.

For the aggregate comparison of all correct and all incorrect rollouts, without further grouping incorrect rollouts by answer identity, Appendix A, Figure 7 already reports the results: the base Qwen3-4B model generates 10 rollouts for each of 1,000 problems, and we compare three types of within-problem pairs at the anchor. Pooling eligible pairs as in the figure gives the means below; Figure 7 provides the full distributions and comparisons across token positions. These are aggregate pair statistics, whereas the 211-problem control above specifically conditions on a shared wrong answer and weights problems equally.

Pair typeMean cosine similarity pooled over pairs
Correct–correct0.854
Correct–wrong0.744
Wrong–wrong0.808

Correct–correct pairs have the highest mean similarity in the aggregate comparison; conditioning on a shared wrong answer yields within-group similarity close to that of correct pairs. One interpretation consistent with these observations is that shared answer information is itself part of a shared decision representation: different reasoning paths can form representations supporting the same answer, whether that answer is correct or incorrect. In autoregressive generation, subsequent answers are predicted from preceding hidden representations, making answer agreement a useful clue to what those representations encode. Hidden-Align uses the verifier to select correct trajectories and then aligns their answer-boundary representations. Our mechanistic hypothesis is that this reduces variation unrelated to the answer decision among successful trajectories, helping different reasoning paths support the correct answer more consistently.

The effectiveness of alignment is tested directly through training comparisons: the paper reports gains across model scales and seeds, and Appendix G finds four-task gains of +7.86 points for hidden-state alignment and +0.89 for logit alignment. This also shows that constraining intermediate representations can provide additional gains beyond alignment at the final output layer.

AuthorsOpenReview ↗

Response to Reviewer NUQ3 (2/3)

W3 / Q3: Checkpoint selection and evaluation overlap

Could you clarify how checkpoints were selected and whether the validation data overlap with the final evaluation benchmarks?

We introduced training-time validation as a lower-cost basis for checkpoint selection because evaluating every checkpoint on all eight benchmarks was computationally expensive, with the aim of avoiding checkpoint selection on the test set. Appendix B reports validation-based selection of step500 for 1.7B DAPO and step520 for 1.7B Hidden-Align. An inadvertent oversight in constructing the archived validation mixture introduced overlap with AIME24, AIME25, AMC23, and MinervaMath in the final evaluation. By question count, these comprise 372 of the 1,151 evaluation questions (32.3%); the other 779 questions come from four benchmarks absent from the validation mixture. The training and evaluation data are separate, and validation scores are not used for gradient updates.

To directly examine whether the main conclusion depends on the overlapping benchmarks, we retain the checkpoints reported in the main table, exclude those four benchmarks, and recompute the results across all model scales and RL algorithms:

ModelMethodAIME26AMC24HMMTOlympiadBenchFour-task averageHA gain (pp)
Qwen3-4BDAPO40.3158.7519.7963.3545.55—
Qwen3-4BDAPO + HA50.5263.4027.8167.3652.27+6.72
Qwen3-4BGSPO39.4857.2921.3564.3945.63—
Qwen3-4BGSPO + HA43.6558.4720.5263.9546.65+1.02
Qwen3-1.7BDAPO19.9047.4314.0649.7032.77—
Qwen3-1.7BDAPO + HA24.2747.5716.9855.3436.04+3.27
Qwen3-14BDAPO49.7964.1031.3572.8554.52—
Qwen3-14BDAPO + HA60.9465.6234.5875.8259.24+4.72

The average gain remains positive for all three model scales and both RL algorithms after excluding the validation-mixture benchmarks. The DAPO gains are 3.27, 6.72, and 4.72 percentage points, and the GSPO gain is 1.02 points. DAPO + Hidden-Align also improves every retained benchmark at all three scales. The performance benefit therefore extends to evaluation sources absent from the validation mixture.

Q4: Anchor and weight choices across seeds

Are the preferred anchor position and alignment weight consistent across training seeds?

We repeated two alternative anchors and two alternative alignment weights with a second training seed. The default configuration achieves the highest score in this comparison, as shown below:

ConfigurationSeed A: four-task avg.Seed B: four-task avg.Seed B: eight-task avg.Default advantage (Seed B, eight tasks, pp)
Default: pre_boxed, λ=0.00149.6046.5555.75—
Answer position (mean over answer-token states)42.9644.8654.42+1.33
pre_boxed_open43.9243.9453.82+1.93
λ=0.000543.7643.3352.74+3.01
λ=0.000142.9142.6652.68+3.07

The four-task average follows Tables 5 and 13: AIME24, AIME26, AMC24, and HMMT, allowing a cross-seed comparison with the original ablations. It differs from the average excluding validation-mixture benchmarks, which replaces AIME24 with OlympiadBench.

The default leads by 1.33–3.07 percentage points on the eight-task average in the additional seed and ranks first on the four-task average in both seeds, providing additional cross-seed support for the anchor and weight choices in the paper.

AuthorsOpenReview ↗

Response to Reviewer NUQ3 (3/3)

W2 (answer format) / Q5: Transfer with the answer-is marker

What might explain the smaller improvement with the natural-language answer is marker?

The transfer experiment treats the answer convention as a complete setting: we fix the marker, verifier, and anchor rule, then compare training with and without Hidden-Align. This tests whether alignment remains effective under another answer convention. In the answer-is setting, adding Hidden-Align raises the eight-task average from 47.45 to 47.98 (+0.53 points), demonstrating applicability to an answer boundary beyond the boxed convention.

For the smaller gain, our hypothesis is that the pre-marker hidden state encodes the preceding reasoning while also drawing on linguistic knowledge acquired during pretraining to predict the phrase introducing the answer. As a common natural-language expression, answer is brings semantic, syntactic, and collocational regularities into that prediction, so the representation may combine reasoning information with language-production information; boxed provides a more explicit structural answer boundary. If alignment mainly brings the existing language-production components closer, its additional contribution to the answer decision may be smaller. This offers an explanation for successful transfer with varying gains and motivates comparing nearby boundary positions when adapting to a new format.

Q6: Additional cost relative to DAPO without monitoring

Could you clarify the additional training cost relative to a DAPO baseline with hidden-state monitoring disabled?

We ran the requested matched comparison with hidden-state monitoring and alignment fully disabled for DAPO and the remaining training settings shared with Hidden-Align. On eight H20D GPUs, over 13 common recorded steps after excluding the initial cold start, reference collection and actor updates added 3.82 seconds per step, equivalent to approximately 1.9% of a DAPO training step.

Both methods took approximately 199 seconds per training step, including rollout generation and excluding periodic validation. Generation lengths vary during training, so we report the alignment-related stage increment separately to show the computation added by the auxiliary objective.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and questions. Hidden-Align uses successful rollouts already available in RLVR to provide additional supervision at the answer boundary. We summarize below the evidence addressing the shared concerns about stability, the role of alignment, transfer, and computational cost.

Are the gains consistent across training seeds? Hidden-Align improves the eight-task average in all three training seeds. The across-seed mean increases from 52.33 ± 1.00 for DAPO to 55.80 ± 2.08 (+3.47 points). Different seeds generate different training rollouts, so the number and content of correct pairs available for alignment also vary. The gain magnitude changes, but the improvement is positive in all three independent runs. We additionally repeated two anchor and two weight ablations with another seed, providing further support for the configuration choices in the paper.

What does alignment contribute to reasoning? Shared answer information can be part of a shared decision representation, while the verifier determines which trajectories to align. Hidden-Align encourages successful trajectories to form representations supporting the correct answer at the answer boundary, without requiring token-by-token matching of intermediate reasoning text. Appendix G reports four-task gains of +7.86 points for hidden-state alignment, compared with +0.89 for logit alignment. Our added text analysis also finds no decrease in overall textual diversity among answer-correct outputs: full-trace distance changes are small and positive in both seeds. Answer-boundary alignment can therefore coexist with varied reasoning expression.

How does the method transfer to new tasks and answer formats? Our anchor-selection principle is to use the representation at the transition from reasoning to final-answer production. The mathematical experiments use the token before the answer marker; spatial reasoning uses the token before </think>. Both apply the same alignment objective. Within the alternative answer is convention, adding Hidden-Align raises the eight-task average from 47.45 to 47.98 (+0.53 points). We hypothesize that the smaller gain reflects the pre-marker state's additional role in predicting an ordinary linguistic phrase using knowledge acquired during pretraining. Alignment may act more on language production in this setting and contribute less directly to the answer decision. For a new format, we recommend identifying the answer boundary and comparing a few plausible positions on a separate development set.

How much computation does alignment add? The implementation collects reference representations during the existing old-policy forward pass, extracts the representations to align during the normal actor-update forward pass, and combines the alignment and RL losses for backpropagation. In matched profiling with hidden-state monitoring and alignment fully disabled for DAPO, reference collection and actor updates added 3.82 seconds per step, equivalent to approximately 1.9% of a DAPO training step. Both methods took approximately 199 seconds per training step, including rollout generation and excluding periodic validation. Generation lengths vary during training, so we report the alignment-related stage increment separately to show the computation added by the auxiliary objective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.