acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Credit Assignment for RLVR on Fused Gromov-Wasserstein Geometry

Abstract

Reinforcement learning with verifiable rewards (RLVR) has been shown to improve the reasoning capability of large language models (LLMs) across diverse reasoning tasks. However, group-based RLVR methods, such as GRPO, assign a uniform advantage to all tokens within rollouts of the same outcome. While existing works refine credit assignment of GRPO based on local signals such as token locations or entropy, they often fail to capture the global semantic novelty of a reasoning behavior relative to the current policy. In this work, we propose a hierarchical credit assignment approach for group-based RLVR methods, called HarA, which identifies and encourages semantically novel reasoning behaviors during RLVR. HarA represents each sampled rollout as a distribution over the hidden states and locations of tokens, and computes the Fused Gromov-Wasserstein (FGW) barycenters of all rollouts with the same outcome, capturing the internal reasoning patterns in the latent space under the current policy. The semantic novelty of a reasoning element can then be measured by its contribution to the FGW distance between the current rollout and the barycenter. While solving the FGW formulation is expensive, we introduce an anchor-guided linearization that turns it into a Wasserstein formulation solvable via the Sinkhorn algorithm efficiently. By reweighing token-level advantage of group-based RLVR methods based on the novelty signals, HarA highlights novel reasoning behaviors at flexible granularities to encourage fine-grained LLM exploration. Extensive experiments across three group-based RLVR methods show that our plug-and-play method effectively enhances the exploration of LLMs, outperforming existing methods across diverse reasoning benchmarks.

Primary area:
reinforcement learning
Keywords:
RLVR, LLM Post-Training, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_WBvvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel credit assignment method HARA, which computes the Fused Gromov-Wasserstein (FGW) barycenters of all rollouts with the same reward of a group, and defines new weight based on rollout's contribution the the FGW distance. For computational efficiency, it proposes an anchor-guided linearization for the non-convex GW term, makes it a convex Wasserstein term. Experimental results show superior performance among baselines.

Critical strengths
  1. This paper proposes a novel credit assignment method HARA.
  2. It proposes an anchor-guided linearization to improve optimization efficiency.
  3. Experimental results demonstrate the effectiveness of HARA.
Critical weaknesses
  1. This paper does not establish that FGW novelty corresponds to useful exploration. HARA assumes that reasoning behaviors farther from the outcome-conditioned barycenter deserve stronger encouragement when correct and weaker penalties when incorrect. However, a large FGW distance may also reflect incoherence, formatting variation, or unproductive reasoning rather than useful exploration. Figure 2 mainly shows that barycenter supports correlate with LLM-annotated reasoning stages, not that high-novelty behaviors are more likely to produce future learning gains.
  2. The "hierarchical" characterization appears stronger than what is actually evaluated. Although the framework is formulated using an arbitrary partition into reasoning spans, the experiments only instantiate two extreme cases: the entire rollout as one span (HARA-seq) or every token as an individual span (HARA-tok). The method does not jointly perform sequence- and token-level credit assignment, nor does it evaluate the intermediate reasoning-span granularity suggested by the hierarchical framing.
  3. The inverse novelty weighting for incorrect trajectories has an unspecified numerical edge case. For incorrect rollouts this paper sets $p=-1$, so the credit is proportional to D−1D^{-1}, where D=∑i∈Sϕk,iD=\sum_{i\in S}\phi_{k,i}. If D=0D=0, Eq. (7) is undefined. Even when DD is very small, the normalized weight can become highly concentrated on a single span, yet no ϵ\epsilon, clipping rule, or other stabilization is specified.
  4. The method does not specify how outcome-degenerate groups are handled. HARA constructs separate barycenters for correct and incorrect rollouts, but with only K=16K=16 samples per prompt, it is possible that all sampled rollouts receive the same binary reward, leaving one outcome-conditioned group empty. This paper does not explain how the corresponding barycenter or novelty credit is defined in this case.
  5. Important implementation details of the latent representation are missing. Each rollout is represented using a token hidden-state matrix HH, but this paper does not specify from which Transformer layer these hidden states are extracted.
  6. The model referred to as "GPT-5-Sol" in line 206 and lines 821–822 does not exist.
  7. This paper does not provide the concrete reward rule used in the experiments, and the code is unavailable, which raises a reproducibility concern.
Questions for authors

Please refer to the Weaknesses.

Reviewer_9PLAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the issue that GRPO assigns the same advantage to every token in a rollout. The authors note that existing methods reweight tokens by position or entropy, but these are local signals within a single response and cannot tell whether a reasoning segment takes a different approach from the other responses in the group. The paper proposes HARA, which represents each rollout as a distribution over its tokens’ hidden states and positions. For each prompt, HARA computes an FGW barycenter for the correct and the incorrect responses separately, measures each token’s novelty by its distance to the barycenter after alignment, and reweights the GRPO advantages accordingly. To reduce cost, the authors linearize FGW into a Wasserstein problem solvable with Sinkhorn, using the first and last tokens as anchors, and provide a bound on the approximation error. Trained on MATH with Qwen3-1.7B/4B/8B Base, HARA improves over GRPO by 3.7 to 4.8 points on average across five math benchmarks, with 5% to 14% extra training overhead.

Critical strengths
  1. The problem is meaningful. Coarse-grained credit assignment caused by assigning a uniform advantage within each group is indeed a pain point of GRPO. Using “distribution-level novelty relative to the current policy” for credit assignment is an intuitive idea, and the method is plug-and-play.
  2. The paper also reports the training overhead and a time–accuracy comparison against GRPO.
  3. The theoretical derivations appear to be correct.
Critical weaknesses
  1. Limited novelty. The most closely related work, SHEAR, which also uses hidden states and the Wasserstein distance for credit assignment, is mentioned only in a single sentence in the related work section. The paper lacks both a theoretical comparison and an experimental comparison with it.
  2. “Hierarchical” is an overclaim. The claimed “arbitrary granularity” is reflected only in two levels, sequence and token. There is no hierarchical combination across levels, and no segment-level variant is explored.
  3. The baseline comparison is flawed. According to Figure 4 (Qwen3-8B-Base), HARA’s responses are about twice as long as GRPO’s. With a maximum length of 8196 tokens, GRPO averages only about 1k tokens, which suggests the baseline may be under-tuned. Since longer responses generally lead to higher accuracy on math tasks, HARA’s gains may mainly come from longer responses rather than from credit assignment. Moreover, nearly doubling the response length while adding only 5%–14% training overhead is inconsistent with the fact that rollout generation usually dominates training time. I suggest the authors report response lengths for all methods and provide comparisons under comparable lengths.
Questions for authors

See above

Comments on the paper

1 comment

Dear reviewer,

Thank you again for your thoughtful reviews and comments. As the deadline for the rebuttal period approaches, we would like to respectfully inquire whether our responses address your concerns properly, and we are more than happy to address any remaining points. Many thanks for your time and effort!

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.