- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a novel credit assignment method HARA, which computes the Fused Gromov-Wasserstein (FGW) barycenters of all rollouts with the same reward of a group, and defines new weight based on rollout's contribution the the FGW distance. For computational efficiency, it proposes an anchor-guided linearization for the non-convex GW term, makes it a convex Wasserstein term. Experimental results show superior performance among baselines.
Critical strengths
- This paper proposes a novel credit assignment method HARA.
- It proposes an anchor-guided linearization to improve optimization efficiency.
- Experimental results demonstrate the effectiveness of HARA.
Critical weaknesses
- This paper does not establish that FGW novelty corresponds to useful exploration. HARA assumes that reasoning behaviors farther from the outcome-conditioned barycenter deserve stronger encouragement when correct and weaker penalties when incorrect. However, a large FGW distance may also reflect incoherence, formatting variation, or unproductive reasoning rather than useful exploration. Figure 2 mainly shows that barycenter supports correlate with LLM-annotated reasoning stages, not that high-novelty behaviors are more likely to produce future learning gains.
- The "hierarchical" characterization appears stronger than what is actually evaluated. Although the framework is formulated using an arbitrary partition into reasoning spans, the experiments only instantiate two extreme cases: the entire rollout as one span (HARA-seq) or every token as an individual span (HARA-tok). The method does not jointly perform sequence- and token-level credit assignment, nor does it evaluate the intermediate reasoning-span granularity suggested by the hierarchical framing.
- The inverse novelty weighting for incorrect trajectories has an unspecified numerical edge case. For incorrect rollouts this paper sets $p=-1$, so the credit is proportional to , where . If , Eq. (7) is undefined. Even when is very small, the normalized weight can become highly concentrated on a single span, yet no , clipping rule, or other stabilization is specified.
- The method does not specify how outcome-degenerate groups are handled. HARA constructs separate barycenters for correct and incorrect rollouts, but with only samples per prompt, it is possible that all sampled rollouts receive the same binary reward, leaving one outcome-conditioned group empty. This paper does not explain how the corresponding barycenter or novelty credit is defined in this case.
- Important implementation details of the latent representation are missing. Each rollout is represented using a token hidden-state matrix , but this paper does not specify from which Transformer layer these hidden states are extracted.
- The model referred to as "GPT-5-Sol" in line 206 and lines 821–822 does not exist.
- This paper does not provide the concrete reward rule used in the experiments, and the code is unavailable, which raises a reproducibility concern.
Questions for authors
Please refer to the Weaknesses.
Reminder
Dear reviewer,
Thank you again for your thoughtful reviews and comments. As the deadline for the rebuttal period approaches, we would like to respectfully inquire whether our responses address your concerns properly, and we are more than happy to address any remaining points. Many thanks for your time and effort!
Best, Authors