- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a novel method called geometric base hidden state trajectory credit allocation, aiming to address the problem of overly coarse-grained credit allocation in verifiable reward reinforcement learning. This method does not rely on expensive process-level supervision or computationally intensive Monte Carlo sampling. Instead, it directly utilizes the hidden state trajectories within the model. By introducing the dynamic time warping algorithm to align inference trajectories of different lengths and quantifying the step-level geometric divergence in a low-dimensional space, this method can effectively re-weight sequence-level advantages. Experimental results show that this method outperforms existing baselines in multiple model scales and achieves dual optimization of performance and computational efficiency in complex mathematical reasoning benchmark tests.
Critical strengths
-
The current reinforcement learning for large language model inference indeed faces the problem of uneven reward distribution at the sequence level, and obtaining high-quality process-level annotations is extremely costly. The unsupervised internal trajectory analysis approach proposed in this paper has high practical value.
-
Using dynamic time warping to align trajectories of different lengths and inference rhythms is a very insightful technical choice, effectively avoiding the noise interference caused by traditional forced step-by-step alignment.
-
Tests were conducted on challenging mathematical reasoning benchmarks, and not only did they report performance improvements, but they also provided detailed comparisons of computational costs, proving that this method improves performance while maintaining extremely high computational efficiency.
Critical weaknesses
-
The paper abstract mentions measuring divergence in the low-dimensional representation space, but does not specify the specific dimensionality reduction method or feature extraction mechanism. This may affect the method's reproducibility.
-
The computational complexity of the dynamic time warping algorithm may have potential issues under extremely long inference trajectories. Although the abstract claims the computational cost can be negligible, complex inference chains involving thousands of tokens may still be affected by the quadratic time complexity of sequence alignment; more detailed ablation experiments or theoretical analysis is needed to support this claim.
-
The sensitivity to early divergence of erroneous trajectories needs to be explored. If the model deviates significantly in the early stage of inference, whether dynamic time warping will cause distorted credit allocation in all subsequent steps, this point is lacking clear response strategies in the current abstract description.
-
The article lacks discussion of some new fine-grained reward articles in this field, such as:
[a]. Unlocking Token Rewards via Training-Free Reward Attribution
[b]. Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization in LLM Reasoning
Questions for authors
See weakness. I will raise my score if my concerns were well addressed.
4 comments
Rebuttal Part 2
W3. The sensitivity to early divergence of erroneous trajectories needs to be explored. If the model deviates significantly in the early stage of inference, whether dynamic time warping will cause distorted credit allocation in all subsequent steps, this point is lacking clear response strategies in the current abstract description.
We appreciate this question. We clarify that GAIT incorporates several design choices to mitigate the potential impact of early divergence on subsequent credit assignment.
- Temporal divergence rather than absolute divergence. As defined in Equations (8) and (9), GAIT reweights step-level advantages based on the temporal change in normalized divergence: Consequently, a large divergence at an early step does not necessarily lead to large advantage adjustments at subsequent steps.
- Global DTW alignment. As defined in Equation (4), DTW identifies an alignment that minimizes the total step-level matching cost between two trajectories. By optimizing alignment globally rather than relying on independent local matches, DTW prevents early divergences from distorting the alignment of subsequent steps.
- Normalization and clipping. GAIT further incorporates divergence normalization (Equation (7)) and clipped advantage reweighting (Equation (9)). These mechanisms limit the influence of extreme divergence values and help maintain training stability.
Moreover, our additional empirical analysis supports the robustness of GAIT to early divergence. For trajectories exhibiting divergence within the first 30% of reasoning steps, subsequent steps receive, on average, only 0.3%–0.5% greater reweighting compared to subsequent steps in trajectories where divergence occurs within the remaining 70%. Moreover, at most 1% of subsequent steps reach the clipping threshold.
We have revised Section 6 (Lines 499-508) to incorporate this analysis.
W4. The article lacks discussion of some new fine-grained reward articles in this field.
We thank the reviewer for highlighting these relevant studies. We have incorporated these references into the revised Section 2 (Line 126 and Lines 130-135).
Specifically, P2T [2] attributes PRM rewards to individual tokens using a gradient-based approximation, but still requires an existing PRM. TRAC [3] derives token-level rewards from teacher-policy log-probability differences and adaptively calibrates them based on teacher reliability at the problem, trajectory, and token levels.
In contrast, our method derives step-level credit from hidden-state trajectories, without requiring a separate reward model, teacher model, or additional MC rollouts.
References
[2] "Unlocking Token Rewards via Training-Free Reward Attribution." CVPR, 2026.
[3] "TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization." ACL, 2026
Thanks. I raised my rate.
Dear Reviewer vyH2,
Thank you for increasing your rating and for your thoughtful engagement throughout the review and discussion process! We are glad that our response has addressed your concerns, and we sincerely appreciate your constructive feedback and support.
Rebuttal Part 1
We sincerely thank the reviewer for recognizing the practical value of our method, the insightful use of dynamic time warping (DTW), and its ability to improve reasoning performance with minimal computational overhead. We also appreciate the constructive feedback. Below, we address each weakness (W) in detail.
We thank the reviewer for raising this point. We would like to clarify that the dimensionality reduction method is already described in Section 4.1, Equation (3), and Algorithm 1. Specifically, we adopt the HARP projection [1], retaining 5% of the original hidden dimensions.
To elaborate, following HARP [1], let W denote the policy model's unembedding (LM head) matrix. We project the hidden states onto the eigenvectors of W⊤W associated with the smallest eigenvalues, retaining 5% of the hidden dimensions. These directions are least sensitive to the unembedding operation, allowing the projection to suppress lexical variations while preserving information in the low-dimensional representation space. Importantly, the projection requires no additional training data or learned parameters and is recomputed from the current model weights at each RL step.
Notably, HARP was originally proposed for hallucination detection rather than RL. Our work adapts this projection to trajectory-based credit assignment. We refer the reviewer to reference [1] for more implementation details.
We have revised Section 4.1 (Lines 234-235) and Appendix C (Lines 960-967) to provide additional details on the implementation of HARP.
References
[1] "HARP: Hallucination Detection via Reasoning Subspace Projection." ICLR, 2026.
We thank the reviewer for this question. We would like to clarify three important points.
First, DTW operates on reasoning steps rather than individual tokens, as described in Section 4.1 and Algorithm 1. Specifically, we segment reasoning trajectories using
\n\ndelimiters and perform DTW alignment at the step level. Although our experiments allow rollouts of up to 12k tokens, the number of reasoning steps is substantially smaller, significantly reducing the alignment cost.Second, our empirical results in Section 5.2 and Figure 8 demonstrate that GAIT introduces minimal computational overhead. Specifically, GAIT on Qwen3-8B-Base takes only 34.5 seconds per RL step on average, accounting for just 3.1% of the average step time.
Third, using HARP projection reduces the hidden state dimensionality to 5% of the original size, significantly lowering the computational cost of pairwise L2 calculations. We also provide an ablation study in Section 5.3 examining the role of this projection and the resulting speedup.
Together, these design choices and empirical results demonstrate that GAIT achieves efficient step-level credit assignment with minimal overhead under our evaluated settings.