- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors study how to turn the single response-level ORM into token-level rewards for RLHF, without training an additional reward model. They propose Single-Source Entropic Optimal Transport (SEOT), which treats the final hidden state of the ORM as a single source and the hidden states of the response tokens as targets, with the transport cost given by their cosine similarity. The entropy-regularized problem has a closed-form solution and the resulting weights split the original reward across tokens while keeping its total unchanged. Experiments on mathematical reasoning and general-domain instruction following compare SEOT with training-based token-level reward models and other redistribution methods.
Critical strengths
The method is simple and cheap to use. SEOT needs only one forward pass of a frozen ORM, with no extra training or iterative optimization, and since the weights sum to one it leaves the original reward untouched. Across both backbones and the general-domain setting, SEOT improves over the other training-free redistribution baselines in Pass@1.
Critical weaknesses
-
The paper is not yet sufficiently polished and feels somewhat repetitive. The same statement, "the outcome reward is predicted from its terminal hidden representation, while intermediate token representations exhibit different degrees of alignment with this terminal state", is restated almost word for word in the abstract, the introduction, and Section 4.1, making the writing feel monotonous. Yet the paper does not convincingly explain why this reward redistribution leads to better policy optimization.
-
The hidden state space of the ORM has no specified geometry, and I could not find a reason why a larger cosine similarity to the final hidden state should mean a larger contribution to the preference judgment. The reward depends on the hidden states only through .
-
The hypothesis on the hidden states is not supported by a direct experiment. For example, if the critical token of a ground truth answer is replaced with a wrong one, the cosine similarity of that token to the final hidden state should change noticeably, while descriptive tokens should be largely unaffected.
-
In Figures 4~6, critical tokens do not receive noticeably higher weights than the other tokens. The weights stay within a narrow range, which reads as the reward being spread almost uniformly by the softmax. A uniform redistribution baseline is missing, so it is unclear whether the gains come from the hidden state similarity or simply from spreading the reward evenly. This is my main concern, and I would be willing to raise my score depending on the authors' explanation of this point.
Minor issues
- Line 344: BT-ORM is not defined.
- denotes both the trajectory and the entropy coefficient (e.g., Eq. 5 and Algorithm 1).
- Figure 4 in Appendix D would help the reader if moved to the main text.
- Line 352: "three types of reward distribution methods" "three types of reward redistribution methods"
Questions for authors
-
Could the authors report results with uniform redistribution under the same GAE-free PPO setting?
-
When the critical token of a ground truth answer is replaced with an incorrect one, how much does its cosine similarity to the final hidden state change compared with that of descriptive tokens?