acceptodds
Under review as a conference paper at ICLR 2027

GAIT: Geometry-Based Credit Assignment from Hidden-State Trajectories for RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a popular approach for improving LLM reasoning. However, credit assignment in mainstream RLVR methods remains coarse-grained: a sequence-level reward is typically propagated across the entire response, regardless of which reasoning steps are essential for the final outcome. Existing approaches to step-level credit assignment often rely on additional supervision that is costly to obtain, such as process reward models, or computationally expensive Monte Carlo sampling that branches from intermediate steps. In this work, we propose **G**eometry-based Credit **A**ssignment from H**I**dden-state **T**rajectories (GAIT), a plug-and-play method that derives step-level credits directly from the model's hidden-state trajectories, without requiring any external step-level supervisor. To compare the geometry of responses with different lengths and reasoning paces, GAIT aligns each response with successful responses from the same rollout group using dynamic time warping, an algorithm for matching temporal sequences that may differ in length. It then measures step-wise divergence in a low-dimensional representation space, and uses temporal changes in this divergence to reweight the sequence-level advantage across reasoning steps. Intuitively, this assigns stronger negative updates to steps in incorrect responses that increasingly diverge from successful trajectories, and stronger positive updates to atypical but successful transitions in correct responses. Across model scales and families, GAIT outperforms existing RLVR baselines with and without fine-grained credit assignment. On challenging AIME 2024-2026 benchmarks with Qwen3-8b-Bases as the base model, GAIT improves over DAPO by 2.38 points and reaches DAPO's peak performance with 1.31 fewer GPU hours, while adding negligible computational overhead to each RL update.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Credit Assignment, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vyH2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel method called geometric base hidden state trajectory credit allocation, aiming to address the problem of overly coarse-grained credit allocation in verifiable reward reinforcement learning. This method does not rely on expensive process-level supervision or computationally intensive Monte Carlo sampling. Instead, it directly utilizes the hidden state trajectories within the model. By introducing the dynamic time warping algorithm to align inference trajectories of different lengths and quantifying the step-level geometric divergence in a low-dimensional space, this method can effectively re-weight sequence-level advantages. Experimental results show that this method outperforms existing baselines in multiple model scales and achieves dual optimization of performance and computational efficiency in complex mathematical reasoning benchmark tests.

Critical strengths
  1. The current reinforcement learning for large language model inference indeed faces the problem of uneven reward distribution at the sequence level, and obtaining high-quality process-level annotations is extremely costly. The unsupervised internal trajectory analysis approach proposed in this paper has high practical value.

  2. Using dynamic time warping to align trajectories of different lengths and inference rhythms is a very insightful technical choice, effectively avoiding the noise interference caused by traditional forced step-by-step alignment.

  3. Tests were conducted on challenging mathematical reasoning benchmarks, and not only did they report performance improvements, but they also provided detailed comparisons of computational costs, proving that this method improves performance while maintaining extremely high computational efficiency.

Critical weaknesses
  1. The paper abstract mentions measuring divergence in the low-dimensional representation space, but does not specify the specific dimensionality reduction method or feature extraction mechanism. This may affect the method's reproducibility.

  2. The computational complexity of the dynamic time warping algorithm may have potential issues under extremely long inference trajectories. Although the abstract claims the computational cost can be negligible, complex inference chains involving thousands of tokens may still be affected by the quadratic time complexity of sequence alignment; more detailed ablation experiments or theoretical analysis is needed to support this claim.

  3. The sensitivity to early divergence of erroneous trajectories needs to be explored. If the model deviates significantly in the early stage of inference, whether dynamic time warping will cause distorted credit allocation in all subsequent steps, this point is lacking clear response strategies in the current abstract description.

  4. The article lacks discussion of some new fine-grained reward articles in this field, such as:

[a]. Unlocking Token Rewards via Training-Free Reward Attribution

[b]. Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization in LLM Reasoning

Questions for authors

See weakness. I will raise my score if my concerns were well addressed.

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We sincerely thank the reviewer for recognizing the practical value of our method, the insightful use of dynamic time warping (DTW), and its ability to improve reasoning performance with minimal computational overhead. We also appreciate the constructive feedback. Below, we address each weakness (W) in detail.


W1. The paper abstract mentions measuring divergence in the low-dimensional representation space, but does not specify the specific dimensionality reduction method or feature extraction mechanism.

We thank the reviewer for raising this point. We would like to clarify that the dimensionality reduction method is already described in Section 4.1, Equation (3), and Algorithm 1. Specifically, we adopt the HARP projection [1], retaining 5% of the original hidden dimensions.

To elaborate, following HARP [1], let WW denote the policy model's unembedding (LM head) matrix. We project the hidden states onto the eigenvectors of W⊤WW^\top W associated with the smallest eigenvalues, retaining 5% of the hidden dimensions. These directions are least sensitive to the unembedding operation, allowing the projection to suppress lexical variations while preserving information in the low-dimensional representation space. Importantly, the projection requires no additional training data or learned parameters and is recomputed from the current model weights at each RL step.

Notably, HARP was originally proposed for hallucination detection rather than RL. Our work adapts this projection to trajectory-based credit assignment. We refer the reviewer to reference [1] for more implementation details.

We have revised Section 4.1 (Lines 234-235) and Appendix C (Lines 960-967) to provide additional details on the implementation of HARP.

References

[1] "HARP: Hallucination Detection via Reasoning Subspace Projection." ICLR, 2026.


W2. The computational complexity of the dynamic time warping algorithm may have potential issues under extremely long inference trajectories.

We thank the reviewer for this question. We would like to clarify three important points.

First, DTW operates on reasoning steps rather than individual tokens, as described in Section 4.1 and Algorithm 1. Specifically, we segment reasoning trajectories using \n\n delimiters and perform DTW alignment at the step level. Although our experiments allow rollouts of up to 12k tokens, the number of reasoning steps is substantially smaller, significantly reducing the alignment cost.

Second, our empirical results in Section 5.2 and Figure 8 demonstrate that GAIT introduces minimal computational overhead. Specifically, GAIT on Qwen3-8B-Base takes only 34.5 seconds per RL step on average, accounting for just 3.1% of the average step time.

Third, using HARP projection reduces the hidden state dimensionality to 5% of the original size, significantly lowering the computational cost of pairwise L2 calculations. We also provide an ablation study in Section 5.3 examining the role of this projection and the resulting speedup.

Together, these design choices and empirical results demonstrate that GAIT achieves efficient step-level credit assignment with minimal overhead under our evaluated settings.

AuthorsOpenReview ↗

Rebuttal Part 2

W3. The sensitivity to early divergence of erroneous trajectories needs to be explored. If the model deviates significantly in the early stage of inference, whether dynamic time warping will cause distorted credit allocation in all subsequent steps, this point is lacking clear response strategies in the current abstract description.

We appreciate this question. We clarify that GAIT incorporates several design choices to mitigate the potential impact of early divergence on subsequent credit assignment.

  • Temporal divergence rather than absolute divergence. As defined in Equations (8) and (9), GAIT reweights step-level advantages based on the temporal change in normalized divergence: Δd^i,t=d^i,t−d^i,t−1.\Delta \hat{d}_{i,t}=\hat{d}_{i,t}-\hat{d}_{i,t-1}. Consequently, a large divergence at an early step does not necessarily lead to large advantage adjustments at subsequent steps.
  • Global DTW alignment. As defined in Equation (4), DTW identifies an alignment that minimizes the total step-level matching cost between two trajectories. By optimizing alignment globally rather than relying on independent local matches, DTW prevents early divergences from distorting the alignment of subsequent steps.
  • Normalization and clipping. GAIT further incorporates divergence normalization (Equation (7)) and clipped advantage reweighting (Equation (9)). These mechanisms limit the influence of extreme divergence values and help maintain training stability.

Moreover, our additional empirical analysis supports the robustness of GAIT to early divergence. For trajectories exhibiting divergence within the first 30% of reasoning steps, subsequent steps receive, on average, only 0.3%–0.5% greater reweighting compared to subsequent steps in trajectories where divergence occurs within the remaining 70%. Moreover, at most 1% of subsequent steps reach the clipping threshold.

We have revised Section 6 (Lines 499-508) to incorporate this analysis.


W4. The article lacks discussion of some new fine-grained reward articles in this field.

We thank the reviewer for highlighting these relevant studies. We have incorporated these references into the revised Section 2 (Line 126 and Lines 130-135).

Specifically, P2T [2] attributes PRM rewards to individual tokens using a gradient-based approximation, but still requires an existing PRM. TRAC [3] derives token-level rewards from teacher-policy log-probability differences and adaptively calibrates them based on teacher reliability at the problem, trajectory, and token levels.

In contrast, our method derives step-level credit from hidden-state trajectories, without requiring a separate reward model, teacher model, or additional MC rollouts.

References

[2] "Unlocking Token Rewards via Training-Free Reward Attribution." CVPR, 2026.

[3] "TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization." ACL, 2026

Thanks. I raised my rate.

Dear Reviewer vyH2,

Thank you for increasing your rating and for your thoughtful engagement throughout the review and discussion process! We are glad that our response has addressed your concerns, and we sincerely appreciate your constructive feedback and support.

Reviewer_GbaZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GAIT, a method for step-level advantage reweighting in reinforcement learning with verifiable rewards. GAIT represents reasoning steps using projected hidden states, aligns each response with successful responses from the same rollout group using dynamic time warping, and computes changes in normalized divergence from these references. These changes determine clipped multipliers on the sequence-level advantage, amplifying updates at divergent steps in both successful and unsuccessful responses.

The method is integrated into DAPO and evaluated on five mathematical reasoning benchmarks using Qwen3-8B-Base, Qwen3-4B-Base, and Llama3.1-8B-Instruct, with three training seeds. GAIT obtains the highest reported average Pass@1 and Pass@32 for each model. On Qwen3-8B-Base, it improves average AIME accuracy by 2.38 percentage points over DAPO, with a reported credit-assignment overhead of 3.1% of training-step time. Ablations and Monte Carlo continuation diagnostics investigate the usefulness of the geometric signal.

Critical strengths
  1. A practical way to exploit internal representations for fine-grained RL updates. GAIT uses successful trajectories already present in the rollout group to construct step-specific learning signals, avoiding a separate process reward model or additional continuation sampling during training. Combining trajectory alignment with changes in representation-space divergence is a useful methodological contribution. The reported 3.1% overhead makes the approach practically attractive.

  2. Empirical support beyond a single model or aggregate comparison. The experiments include three training seeds, multiple model sizes and families, and comparisons with sequence-, step-, and token-level baselines. The random-permutation ablation is particularly informative: disrupting the correspondence between divergence signals and reasoning steps reduces average AIME accuracy from 28.83 to 26.60, close to DAPO’s 26.45. Together with the positive-only and negative-only ablations, this provides evidence that the placement of the reweighting signal matters, rather than the gains arising solely from adding another training component.

Critical weaknesses
  1. The evidence for meaningful step-level credit is suggestive but does not fully isolate the proposed mechanism. The continuation diagnostics condition on whether the complete sampled response succeeds or fails. Consequently, the observed direction of continuation-value changes may partly reflect outcome conditioning rather than the additional information supplied by geometric divergence. The response-ranking diagnostics also use a divergence signal constructed from known-correct references, so they should be explicitly described as label-assisted diagnostics rather than independent correctness prediction. The random-permutation ablation is useful, but comparisons with outcome- and position-matched steps would better establish whether divergence identifies consequential reasoning transitions beyond these simpler factors.

  2. The robustness of the smaller gains and several evaluation details remain unclear. Average Pass@1 improvements over DAPO decrease to 0.78 points on Qwen3-4B and 0.50 points on Llama3.1-8B; the latter is smaller than GAIT’s reported across-run standard deviation of 0.73. The paper selects the better result from two decoding temperatures and presents layer and reweighting-strength sweeps on the evaluation benchmarks, without clearly specifying an independent selection protocol. Ablations also omit uncertainty estimates. These issues do not invalidate the main Qwen3-8B result, but they limit confidence in the strength of the cross-model generalization claims.

Questions for authors
  1. Clarification of evaluation and model selection: Which data were used to select the hidden-state layer, reweighting strength, decoding temperature, and checkpoint? Are the main tables based on the final training checkpoint or a selected checkpoint? Please also specify how Pass@32 was estimated from the stated 32 samples per prompt per temperature, including whether samples were pooled across temperatures.

  2. Interpretation of the geometric signal: In the Monte Carlo continuation analysis, do steps with positive divergence show larger changes in continuation accuracy than randomly selected steps matched for response outcome and relative position? Please provide the number of evaluated steps, continuation samples per prefix, and uncertainty estimates. This would help distinguish useful step localization from effects of conditioning on the final response outcome.

  3. Clarification of implementation and applicability: How is the HARP projection constructed and updated during RL, and how is a zero or very small standard deviation handled in Equation (7)? What fraction of training groups satisfies the requirement of at least four correct responses, for each model and over training? This determines how broadly GAIT contributes beyond the DAPO fallback.

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We thank the reviewer for recognizing GAIT's methodological contribution, practical efficiency, comprehensive experiments, and informative ablation studies. We also appreciate the constructive feedback. Below, we address each weakness (W) and question (Q) in detail.


W1.1. The response-ranking diagnostics also use a divergence signal constructed from known-correct references, so they should be explicitly described as label-assisted diagnostics rather than independent correctness prediction.

We thank the reviewer for this suggestion. We agree that the response-ranking analysis in Table 3 is a label-assisted diagnostic, rather than an independent correctness predictor. We have revised Appendix B.1 (Lines 746-748) to explicitly clarify this distinction.


W1.2. The random-permutation ablation is useful, but comparisons with outcome- and position-matched steps would better establish whether divergence identifies consequential reasoning transitions beyond these simpler factors.

We thank the reviewer for this suggestion. We would like to clarify that our random-permutation ablation in Table 2 already controls for response outcomes, while our additional analysis indicates that temporal divergence has little correlation with step position.

  • Outcome. In Section 5.3 and Table 2, we randomly permute temporal divergence values within each individual rollout, rather than across rollouts. Since all steps within a rollout share the same final outcome, this ablation preserves the outcome while disrupting the correspondence between divergence signals and reasoning steps. Therefore, the current ablation study already supports the importance of step-specific divergence information beyond the final response outcome.
  • Position. GAIT uses temporal divergence for advantage reweighting (Equation (9)), rather than the normalized divergence in Equation (7). For analysis, we computed the Spearman correlation between temporal divergence and relative step position. The correlations range from 0.032 to 0.068 for Qwen3-8B-Base, Qwen3-4B-Base, and Llama3.1-8B-Instruct, and from 0.007 to 0.015 after GAIT training. These near-zero correlations suggest that temporal divergence is not driven by step position.

We have revised Section 5.3 (Lines 460-462) to clarify the outcome-preserving nature of our ablation and used the discussion in Section 6 (Lines 493-498) to address the concern.

AuthorsOpenReview ↗

Rebuttal Part 2

W2.1. The paper selects the better result from two decoding temperatures and presents layer and reweighting-strength sweeps on the evaluation benchmarks, without clearly specifying an independent selection protocol.

We thank the reviewer for this question. We clarify that the same evaluation protocol is applied to all methods to ensure consistent comparisons. We provide the detailed protocol in our response to Q1 below.


W2.2. Ablations also omit uncertainty estimates.

We thank the reviewer for the suggestion. We clarify that all ablation experiments were conducted with three independent random seeds.

We have revised Tables 2 and 5 to include uncertainty estimates.


W2.3. The robustness of the smaller gains and several evaluation details remain unclear. Average Pass@1 improvements over DAPO decrease to 0.78 points on Qwen3-4B and 0.50 points on Llama3.1-8B; the latter is smaller than GAIT’s reported across-run standard deviation of 0.73

We thank the reviewer for raising this question. We (i) highlight GAIT's Pass@32 improvements, (ii) discuss a potential explanation for the smaller Pass@1 gains, and (iii) provide new results on Qwen3-14B-Base to further evaluate GAIT's effectiveness.

Pass@32.

We acknowledge that GAIT achieves smaller Pass@1 improvements on Qwen3-4B-Base and Llama3.1-8B-Instruct than on Qwen3-8B-Base. However, as reported in Section 5.2 and Table 4, GAIT also improves Pass@32 by 2.89 and 0.85 points on these two models, respectively, exceeding their corresponding Pass@1 gains. These results provide additional evidence of GAIT's effectiveness across different model architectures and sizes.

Discussion.

We hypothesize that the smaller Pass@1 gains may be partly attributable to GAIT's fallback mechanism for challenging training prompts. As described in Section 5.1 and Appendix C, GAIT requires at least four correct rollouts per training prompt to compute meaningful normalized divergence signals (Equation (7)). When fewer than four correct rollouts are available, GAIT falls back to the original DAPO advantage. Consequently, extremely challenging training prompts with very few correct rollouts receive no additional step-level credit assignment, making GAIT's updates closer to those of DAPO. As detailed in our response to Q3.3, this requirement is satisfied less frequently for Llama3.1-8B-Instruct than for the Qwen3 models, which may partly explain its smaller improvement.

Qwen3-14B-Base Results.

To further evaluate GAIT's generalizability, we conduct additional experiments on Qwen3-14B-Base. Due to computational and time constraints, we run DAPO and GAIT once. GAIT outperforms DAPO on all five benchmarks, achieving a 2.22-point improvement in average Pass@1.

MethodAIME24AIME25AIME26AMC23BeyondAIMEAvg.
DAPO47.1933.2336.4684.9218.6944.10
GAIT (Ours)47.7136.0439.7987.4220.6646.32

We have revised the manuscript to include an additional discussion of this in Section 6 (Lines 485-492) and added the Qwen3-14B-Base results to Appendix D.2.

AuthorsOpenReview ↗

Rebuttal Part 3

Q1: Clarification of evaluation and model selection: Which data were used to select the hidden-state layer, reweighting strength, decoding temperature, and checkpoint? Are the main tables based on the final training checkpoint or a selected checkpoint? Please also specify how Pass@32 was estimated from the stated 32 samples per prompt per temperature, including whether samples were pooled across temperatures.

We thank the reviewer for these questions. We clarify each aspect of our evaluation protocol below. Importantly, the same evaluation and checkpoint-selection procedures are applied to all compared methods.


Q1.1. Hidden-state layer.

As described in Section 5.1 and Appendix C, we consistently select the hidden-state layer at 90% of the model depth, regardless of the model architecture. This choice is fixed and does not depend on data.


Q1.2. Reweighting strength.

As shown in the ablation study in Section 5.3 and Figure 10, we select the reweighting strength based on average performance on AIME 2024, 2025, and 2026.


Q1.3. Decoding temperature.

As described in Section 5.1, we evaluate each method on each benchmark at two decoding temperatures, 0.6 and 1.0, and report the higher of the two results. This procedure is applied consistently across all compared methods.


Q1.4. Checkpoint selection.

As described in the captions of Figures 1 and 2, we select the checkpoint that achieves the highest average performance across AIME 2024, 2025, and 2026 for each method.


Q1.5. Pass@32 computation.

For each model, benchmark, and decoding temperature, we independently sample 64 responses per prompt and estimate Pass@32 using these samples. We compute Pass@32 separately for temperatures 0.6 and 1.0 and report the higher value, consistent with our Pass@1 evaluation protocol. Samples from different temperatures are not pooled.

To provide additional details on the evaluation, we have revised Section 5.1 (Lines 409-410) and Appendix C (Line 999 and Lines 1003-1009).


Q2. Interpretation of the geometric signal: In the Monte Carlo continuation analysis, do steps with positive divergence show larger changes in continuation accuracy than randomly selected steps matched for response outcome and relative position? Please provide the number of evaluated steps, continuation samples per prefix, and uncertainty estimates. This would help distinguish useful step localization from effects of conditioning on the final response outcome.

We thank the reviewer for this question. We first clarify the distinct roles of positive temporal divergence in correct and incorrect rollouts, and then provide an additional outcome- and position-matched experiment.

Positive temporal divergence in correct rollouts. As discussed in Intuition 2 (Section 4.2.1), positive temporal divergence may reflect alternative reasoning strategies in representation space. Accordingly, GAIT encourages such steps through advantage reweighting (Equation (9)). Our hypothesis does not require these steps to have larger changes in continuation accuracy relative to random steps. Instead, it requires such divergence not to harm continuation accuracy, consistent with Figure 7.

Positive temporal divergence in incorrect rollouts. We conduct additional matched-control experiments to examine whether these steps exhibit larger decreases in continuation accuracy than random steps.

Experimental setup.

For each model, we randomly sample 800 steps with positive temporal divergence from incorrect rollouts associated with 200 training prompts. We then sample comparison steps matched by response outcome and relative step position. For each prefix, we generate 16 continuations to estimate continuation accuracy.

We report the difference in continuation-accuracy change between positive-divergence steps and matched random steps:

ModelDifference in MC Accuracy Change (%)
Qwen3-8B-Base−0.41±0.29-0.41 \pm 0.29
+ DAPO−0.62±0.45-0.62 \pm 0.45
Llama3.1-8B-Instruct−0.07±0.25-0.07 \pm 0.25
+ DAPO−0.81±0.43-0.81 \pm 0.43

A negative difference indicates that positive-temporal-divergence steps exhibit a larger decrease in continuation accuracy than outcome- and position-matched random steps. We observe negative differences across all four model settings. Although the difference is small for Llama3.1-8B-Instruct before RL, it becomes substantially more negative through DAPO training (from −0.07 to −0.81).

These results provide additional evidence that temporal divergence captures information about consequential reasoning transitions beyond final response outcomes and relative step positions, supporting its use for fine-grained credit assignment in GAIT.

We have added this analysis to Appendix B.3.

AuthorsOpenReview ↗

Rebuttal Part 4

Q3. Clarification of implementation and applicability: How is the HARP projection constructed and updated during RL, and how is a zero or very small standard deviation handled in Equation (7)? What fraction of training groups satisfies the requirement of at least four correct responses, for each model and over training? This determines how broadly GAIT contributes beyond the DAPO fallback.

We thank the reviewer for these questions. We address each implementation detail below.


Q3.1. HARP projection construction and updating.

Following HARP [1], let WW denote the policy model's unembedding (LM head) matrix. We project the hidden states onto the eigenvectors of W⊤WW^\top W associated with the smallest eigenvalues, retaining 5% of the hidden dimensions. These directions are least sensitive to the unembedding operation, allowing the projection to suppress lexical variations while preserving information in the low-dimensional representation space. Importantly, the projection requires no additional training data or learned parameters and is recomputed from the current policy weights at each RL step.

Notably, HARP was originally proposed for hallucination detection rather than RL. Our work adapts this projection to trajectory-based credit assignment. We refer the reviewer to reference [1] for more implementation details.

We have revised Section 4.1 (Lines 234-235) and Appendix C (Lines 960-967) to provide additional details on the implementation of HARP.

References

[1] "HARP: Hallucination Detection via Reasoning Subspace Projection." ICLR, 2026


Q3.2. Handling zero or very small standard deviations.

As described in Section 5.1 and Appendix C, GAIT applies step-level reweighting only to rollout groups containing at least four correct rollouts, ensuring sufficient samples for estimating the standard deviation in Equation (7).

For numerical stability, we add a small constant (10−610^{-6}) to the denominator to prevent division by zero or excessively small values.

Furthermore, the clipping operation in Equation (9) bounds the reweighting factor, preventing large normalized divergence values from destabilizing training.

We have revised Appendix C (Lines 967-970) to clarify this point.


Q3.3. Fraction of eligible training groups.

The following table reports the average proportion of training rollout groups containing at least four correct responses, measured throughout training after DAPO's dynamic filtering.

Base ModelEligible Groups (%)
Qwen3-8B-Base82.62
Qwen3-4B-Base79.04
Llama3.1-8B-Instruct69.96

These results demonstrate that GAIT's step-level credit assignment applies to a substantial majority of training groups across all three models, rather than frequently falling back to the original DAPO advantage.

We have incorporated these results and additional discussion into Section 6 (Lines 485-492) and Table 10 of the revised manuscript.

Reviewer_sPS7OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduced Geometry-based Credit Assignment from HIdden-state Trajectories to address the credit assignment issue in the GRPO-based methods like DAPO. Experimental results indicate the effectiveness of the proposed method.

Critical strengths
  1. The paper is well-written and easy follow, with good diagram for presentation.
  2. The related work is well discussed and the setting, and the task are well defined.
Critical weaknesses
  1. In line 51-74, the authors have discussed the current methods to address the credit assginment issue. And reporting these two categories of methods are "Both introduce substantial overhead in data collection and computation". Is there any experiment to prove this statement? Also the comparasion in table 1: for setting with Qwen3-4B-Base and Llama3.1-8B-Instruct, there shoule be compared with more related baseline methods to show the effectiveness of the proposed method.
  2. Experimental results in table 1, the performance of the proposed method is lower than the DAPO on the AIME25 with Llama3.1-8B-Instruct and BeyondAIME with Qwen3-4B-Base. Could the analysis on the failure mode be added and also explain the reason on the performance degradation.
  3. The proposed method is less motivated. The main insight of the proposed method is around the line 195-200, but there is no experiment to support these two insight. Could the author provide the proportion of the cases in these two suitation and analysis why these two specific kinds of trajectories are important.
  4. Analysis for the trajectories before and after the training is necessary to prove the effectiveness of the proposed method. And fail mode analysis also cares.
Questions for authors

Refer to the weakness part.

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We thank the reviewer for recognizing that our paper is "well-written and easy to follow," with effective illustrations, a thorough discussion of related work, and a well-defined task. We also appreciate the reviewer's feedback. Below, we address each weakness (W) in detail.


W1.1. In line 51-74, the authors have discussed the current methods to address the credit assignment issue. And reporting these two categories of methods are "Both introduce substantial overhead in data collection and computation". Is there any experiment to prove this statement?

We thank the reviewer for this question. We clarify that prior work [1, 2, 3] cited in Section 2 already documented the substantial data collection and computational overhead of PRM-based and tree-based RL methods. We also provided experimental evidence of the overhead of tree-based RL (Figure 2 and Section 5.2). Below, we provide more details.

Process Reward Models (PRMs).

  • Data collection: Training PRMs typically requires collecting step-level supervision from human annotators, LLM judges, or Monte Carlo (MC) sampling. This process is expensive. For example, PURE, a baseline in our experiments, uses the PRM800K [1] dataset to train its PRM. PRM800K contains about 800K human-annotated step-level labels. Another widely used dataset, Math-Shepherd [2], requires about 22 million generated rollouts to construct its step-level supervision. These examples illustrate the substantial annotation or sampling costs associated with data collection for PRM training.
  • Computational overhead: Beyond data collection, PRM-based methods require additional computation to train a separate reward model. This introduces extra training costs before the PRM can be used for credit assignment.

Tree-Based RL.

  • Data collection: Tree-based RL relies on online MC sampling to estimate values of intermediate steps. For example, AttnRL [3], a baseline in our experiments, generates approximately 1.6× as many rollouts per RL step as DAPO and GAIT, introducing substantial overhead for collecting data.
  • Computational overhead: As shown in Figure 2, AttnRL requires 1,067 GPU hours to reach its peak performance, approximately 2.89× the GPU hours required by GAIT. Moreover, GAIT achieves higher peak performance with substantially lower computational cost.

These examples and experimental results support our claim that existing PRM-based and tree-based credit assignment methods introduce additional data collection and computational overhead. In contrast, GAIT requires neither external step-level supervision nor additional online rollouts, enabling efficient credit assignment within standard RL training.

References

[1] "Let's Verify Step by Step." ICLR, 2024.

[2] "Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations." ACL, 2024.

[3] "Attention as a Compass: Efficient Exploration for Process-Supervised {RL} in Reasoning Models." ICLR, 2026.


W1.2. Also the comparison in table 1: for setting with Qwen3-4B-Base and Llama3.1-8B-Instruct, there should be compared with more related baseline methods to show the effectiveness of the proposed method.

We thank the reviewer for this suggestion. As mentioned in the caption of Table 1, due to the substantial computational cost, our original experiments on Qwen3-4B-Base and Llama3.1-8B-Instruct included only DAPO and FIPO, the strongest baseline on Qwen3-8B-Base.

To provide a more comprehensive comparison, we conduct additional experiments with GRPO, AttnRL, and PURE on Qwen3-4B-Base, and GRPO and AttnRL on Llama3.1-8B-Instruct.

  • AttnRL: Due to its extremely high computational cost and the time limit of the rebuttal, we allocate the same GPU-hour budget as a 200-step DAPO or GAIT run.
  • PURE: Its publicly released PRM uses a Qwen-series tokenizer, making it incompatible with Llama3.1-8B-Instruct.

Due to time and computational constraints, we conduct each additional experiment once. The additional results are shown below:

Base ModelMethodAIME24AIME25AIME26AMC23BeyondAIMEAvg.
Qwen3-4B-BaseGRPO23.1219.1716.0467.348.3426.80
PURE19.1715.0014.3761.647.6923.57
AttnRL22.5020.6216.7765.318.0926.66
GAIT (Ours)23.9221.1818.4768.148.7228.09
Llama3.1-8B-InstructGRPO10.311.252.7134.921.3810.11
AttnRL8.120.941.1530.861.728.56
GAIT (Ours)11.323.062.8140.782.1412.02

These additional results confirm the findings in Table 1: GAIT consistently outperforms all evaluated baselines across both models, achieving the highest average accuracy and the best performance on every benchmark.

We have added Appendix D.1 to present these additional results.

AuthorsOpenReview ↗

Rebuttal Part 2

W2. Experimental results in table 1, the performance of the proposed method is lower than the DAPO on the AIME25. Could the analysis on the failure mode be added and also explain the reason on the performance degradation.

We thank the reviewer for suggesting this analysis. We (i) discuss the observed performance differences and their potential causes, and (ii) provide new results on Qwen3-14B-Base to further evaluate GAIT's effectiveness.

Clarification and Discussion.

We first clarify that the observed performance differences between GAIT and DAPO in these two cases fall within the reported standard deviations. Thus, these results do not provide strong evidence of a performance degradation, and small differences may reflect evaluation variability, such as randomness in decoding.

We further hypothesize that the comparable performance is related to GAIT's fallback mechanism for extremely challenging training prompts, as described in Section 5.1 and Appendix C.

Specifically, GAIT requires at least four correct rollouts for each training prompt to obtain a meaningful normalization in Equation (7). When fewer than four correct rollouts are available, GAIT falls back to the original DAPO advantage. Consequently, particularly challenging training prompts are less likely to benefit from GAIT's additional step-level credit assignment signal, making its training behavior closer to DAPO.

This may help explain the comparable performance on BeyondAIME with Qwen3-4B-Base (DAPO Pass@1: 8.80%) and AIME25 with Llama3.1-8B-Instruct (DAPO Pass@1: 3.54%). Both settings are extremely challenging for their respective models, suggesting that GAIT's additional credit assignment signal may provide limited benefits on similarly difficult problems.

Qwen3-14B-Base Results.

To further evaluate GAIT's generalizability, we conduct additional experiments on Qwen3-14B-Base. Due to computational and time constraints, we run DAPO and GAIT once. GAIT outperforms DAPO on all five benchmarks, achieving a 2.22-point improvement in average Pass@1.

MethodAIME24AIME25AIME26AMC23BeyondAIMEAvg.
DAPO47.1933.2336.4684.9218.6944.10
GAIT (Ours)47.7136.0439.7987.4220.6646.32

We have revised the manuscript to include an additional discussion of this in Section 6 (Lines 485–492) and added the Qwen3-14B-Base results to Appendix D.2.


W3. There is no experiment to support these two insight. Could the author provide the proportion of the cases in these two suitation and analysis why these two specific kinds of trajectories are important.

We thank the reviewer for this question. We would like to clarify that Section 4.2.1 and Appendix B already provide experiments supporting both intuitions, showing consistent divergence patterns across different model families, both before and after RL training.

Regarding the proportion of cases, Equation (9) applies step-level advantage reweighting to all rollouts whose rollout groups contain at least four correct responses. For Qwen3-8B-Base, Qwen3-4B-Base, and Llama3.1-8B-Instruct, 82.6%, 79.04%, and 69.96% of rollout groups satisfy this requirement on average, respectively, after DAPO's dynamic filtering. This indicates that GAIT's step-level credit assignment applies to a substantial proportion of training rollout groups.

Furthermore, our random permutation ablation in Section 5.3 and Table 2 shows that shuffling temporal divergence signals across reasoning steps reduces average Pass@1 to be comparable to DAPO. This provides additional evidence for the effectiveness of GAIT's step-specific credit assignment mechanism.

AuthorsOpenReview ↗

Rebuttal Part 3

W4.1 Analysis for the trajectories before and after the training is necessary to prove the effectiveness of the proposed method.

We thank the reviewer for this suggestion. We conduct additional analyses of reasoning trajectories before and after GAIT training, focusing on (i) average rollout length (in tokens) and (ii) the discriminative ability of temporal divergence.

Following Appendix B.1, we evaluate the ROC-AUC of the score −max⁡t=1TiΔd^i,t-\max_{t=1}^{T_i} \Delta \hat{d}_{i,t} for distinguishing correct from incorrect rollouts, where Δd^i,t\Delta \hat{d}_{i,t} denotes the temporal divergence at reasoning step tt in rollout oio_i.

Base ModelRollout Length (Before)Rollout Length (After)ROC-AUC (Before)ROC-AUC (After)
Qwen3-8B-Base9423,3000.7660.950
Qwen3-4B-Base9774,1870.7430.965
Llama3.1-8B-Instruct1,5043,3350.8860.915

The results reveal two consistent trends across all three models:

  • Longer reasoning trajectories: Average rollout lengths increase after GAIT training, indicating that the trained models generate longer reasoning trajectories.
  • Stronger discriminative ability: ROC-AUC consistently improves after training. This indicates that temporal divergence becomes substantially more informative for distinguishing correct and incorrect rollouts.

These findings provide further empirical evidence supporting the effectiveness of GAIT.

We have incorporated these analyses into the revised manuscript in Appendix E.


W4.2 And fail mode analysis also cares.

We thank the reviewer for this suggestion. As discussed in our response to W2, one potential limitation of GAIT arises from its fallback mechanism on extremely challenging training prompts. When fewer than four correct rollouts are available, GAIT uses the original DAPO advantage rather than its proposed step-level reweighting. As a result, these prompts receive limited benefit from GAIT's additional credit assignment signal.

We refer the reviewer to our response to W2 for more details.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their constructive feedback. We appreciate their recognition of our work's novelty (Reviewers GbaZ and vyH2), practical efficiency (Reviewers GbaZ and vyH2), comprehensive experimental evaluation (Reviewers GbaZ and vyH2), and writing quality (Reviewers sPS7, GbaZ, and vyH2).

Based on the reviews, we have made the following revisions:

  • Added Appendix D to present additional experimental results (Reviewer sPS7: W1.2; Reviewer GbaZ: W2.3).
  • Added Section 6 to discuss GAIT's fallback mechanism (Reviewer sPS7: W2, W4.2; Reviewer GbaZ: W2.3, Q3.3), step-position effects (Reviewer GbaZ: W1.2), and early divergence (Reviewer vyH2: W3).
  • Added Appendix E to analyze reasoning trajectories before and after training (Reviewer sPS7: W4.1).
  • Revised Appendix B.1 to clarify the interpretation of our response-ranking analysis (Reviewer GbaZ: W1.1) and added Appendix B.3 to provide further analysis of continuation accuracy (Reviewer GbaZ: Q2).
  • Clarified the random-permutation ablation in Section 5.3 (Reviewer GbaZ: W1.2) and added uncertainty estimates to Tables 2 and 5 (Reviewer GbaZ: W2.2).
  • Revised Section 5.1 and Appendix C to clarify the evaluation protocol (Reviewer GbaZ: W2.1, Q1) and implementation details (Reviewer GbaZ: Q3.1, Q3.2; Reviewer vyH2: W1).
  • Added two references to Section 2, as suggested by Reviewer vyH2 (W4).

All revisions are highlighted in blue in the updated manuscript for ease of review. We provide detailed responses to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.