- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a token-level reweighting scheme called ARC for improving large language model reasoning through reinforcement learning with verifiable rewards. The scheme treats hidden states as trajectories and, based on local curvature, assigns higher weights to tokens near points where the trajectory direction changes; this design is motivated by the concept of semantic change boundaries. The authors further examine the method’s alignment with inference tokens and its complementarity with token entropy, and report performance improvements on six mathematics benchmarks using Qwen3-8B and Qwen3-14B. While the idea is compelling and the reported improvements are encouraging, the current evidence is insufficient to clearly explain why curvature is effective, nor does it fully demonstrate the reliability of these gains.
Critical strengths
(1) This paper explores a useful representation-level signal for token-level credit attribution. The proposed soft weights are bounded and normalized within each reply, making it relatively straightforward to adapt the model to the DAPO objective.
(2) The experiments were quite comprehensive, covering two Qwen3 model sizes and six mathematical reasoning benchmarks. The authors also set up entropy-based baselines, conducted ablation experiments by shuffling weights, and performed additional diagnostic experiments on Llama.
Critical weaknesses
Major concerns
(1) The theoretical result does not truly establish the proposed connection to credit allocation. Proposition 1 merely provides an upper bound on the second-order variation of a certain smooth readout function, but high curvature does not necessarily imply a meaningful change in reasoning or a greater contribution to the final reward. The geometric argument here serves, at most, to provide motivation for the method; it does not prove that it identifies important tokens.
(2) These experiments are not sufficient to distinguish curvature from simpler hidden-state signals. Although the paper compares entropy-based weights with shuffled weights, it does not examine alternative metrics such as step size, rotation angle or second-order differences. Without these comparisons, it is difficult to determine whether the performance improvement is actually attributable to curvature itself.
(3) Table 1 does not present results for different independent training seeds. Sampling 32 answers per question helps to estimate the evaluation accuracy, but fails to reflect differences between multiple reinforcement learning runs. This is particularly crucial given that the improvements achieved by Qwen3-8B on several benchmarks are already modest. Consequently, the existing results still do not demonstrate whether the reported improvements are stable.
Minor concerns
(1) The paper needs to clarify the specific source of the hidden states used in Eq.(4) to (5), for example, whether they are derived from the residual flow output or the normalised output of a particular module. Different choices may affect the curvature score and reproducibility.
(2) Section 4.5 notes that the recovery length increases rapidly when strong curvature selection is employed, but does not provide statistical data on the length or training curves. If a small-scale quantitative comparison were included, this design trade-off could be better assessed.
Questions for authors
(1) How were the curvature layers and the weight parameter γ = 0.6 selected? Are the main results robust to other layers and weighting strengths?
(2) Could the authors discuss whether other geometric measures of hidden-state changes, such as step length or turning angle, might also be useful for token reweighting?
(3) The curvature-sensitive layer differs between Qwen3 and Llama. Could the authors discuss what might explain this difference?
3 comments
- Changes in Response Length during Training (Weakness 5)
Thank you for raising this point. We will include the changes in response length during training in the appendix of the revised manuscript.
- Differences in Curvature Sensitivity across Model Families (Question 3)
One possible explanation is that the two models differ in architecture and post-training objectives, causing checks, corrections, or transitions to conclusions during reasoning to appear as more pronounced turns in the hidden-state trajectory at different depths. Moreover, the two models generate their own responses. Differences in tokenization and marker usage may also affect the alignment measured within a -token window. The current results show that curvature is associated with reasoning markers in the intermediate layers of both models, but the peak layer needs to be identified separately for each model.
If our response has addressed your concerns, we would appreciate it if you could reconsider your overall assessment of the paper.
We thank the reviewer for recognizing our method and its performance improvements. We address your questions below.
Proposition 1 provides a geometric motivation for curvature. Curvature thus has a clear mathematical connection to rapid changes induced by the trajectory. The definition of the score follows from this connection independently of the marker-word correlations. The upper bound also has a clear limitation. High curvature allows for larger changes, but does not guarantee that they occur. This explains why curvature can identify candidate turning points without directly equating them with contributions to correctness. We further test the usefulness of ARC in controlled training experiments.
We use two levels of evidence. Analyses of reasoning markers, position, and overlap with entropy characterize the scoring signal. Tables 1 and 3 directly test its usefulness when incorporated into reinforcement learning updates. In particular, Entropy-Soft uses exactly the same soft-weighting mechanism and changes only the scoring signal. ARC scores higher in 11 of the 12 reported model--benchmark combinations. This comparison directly addresses the explanation that the gains come entirely from soft weighting.
In addition, the diagnostic of curvature-only positions examines the extent of overlap with entropy, and the performance drop after shuffling the weights tests the role of matching scores to token positions. These controls support the practical value of representation geometry.
Our method derives a theoretical upper bound on semantic change based on curvature. The derivation determines our choice of score. We discuss the two proposed metrics from this theoretical perspective.
\textbf{Step length cannot distinguish straight motion from turning.} For example, two trajectory segments can have exactly the same adjacent step lengths, with one continuing in a straight line and the other turning through 90∘. Step length assigns them the same score, whereas ARC gives near-zero curvature to the straight segment and positive curvature to the turn.
\textbf{Turning angle captures directional change but lacks an arc-length scale.} The curvature estimate in Eq. (5) is \begin{equation*} \widehat{\kappa}_t \approx 2\sin(\theta_t/2)/\bar{\ell}_t, \end{equation*} where θt is the angle between adjacent displacements and ℓˉt is the mean step length. If two positions have the same turning angle but one completes the turn over half the distance of the other, the turning angle alone assigns them the same score, while ARC's curvature estimates differ by a factor of two.
Thus, step length omits the direction change, and turning angle omits the distance over which that change occurs. ARC retains both pieces of information and directly corresponds to the geometric quantity that bounds the trajectory-bending term in Proposition 1.
We clarify that representation geometry differs across layers. Our experiments show that, at particular layers, curvature represents semantic transitions better than entropy and random selection. The reasoning process itself determines the relevant layers, since only the geometry at certain layers captures this signal. We then use this optimal signal to improve the model's subsequent behavior.
Table compares the two-run mean and the γ=1 results with the original DAPO and ARC results from Table 1.
We compute curvature from the residual stream after the intermediate decoder block. Equation (4) normalizes the displacement vectors between adjacent hidden states to construct unit tangents.