acceptodds
Under review as a conference paper at ICLR 2027

Preference-Aware Reward Redistribution for Token-Level RLHF

Abstract

Outcome reward models (ORMs) provide robust preference signals for reinforcement learning from human feedback (RLHF), yet their scalar response-level rewards leave the contribution of individual token decisions unspecified, creating a granularity mismatch with token-level policy optimization. We observe that the ORM itself provides a natural source of weak supervision for resolving the resulting ambiguity in mapping response-level rewards to individual tokens: the outcome reward is predicted from its terminal hidden representation, while intermediate token representations exhibit different degrees of alignment with this terminal state. We exploit this representational relation as a proxy for token-level preference relevance and propose Single-Source Entropic Optimal Transport (SEOT), a training-free framework that formulates outcome-to-token reward redistribution as a globally normalized allocation problem. SEOT models the terminal hidden state as a Dirac source and response-token hidden states as target supports, with their representational similarity defining preference-aware transportation costs. The resulting single-source entropic formulation admits a closed-form solution and requires neither additional reward model training nor iterative OT optimization, while exactly preserving the original outcome reward. Experiments on mathematical reasoning and general-domain tasks demonstrate consistent improvements over existing reward redistribution methods. Further analysis shows that SEOT better emphasizes preference-critical tokens while avoiding overly concentrated reward allocation, providing more effective token-level guidance for policy optimization.

Primary area:
reinforcement learning
Keywords:
RLHF, Reward Modeling, Token-Level Credit Assignment, Reward Redistribution, Optimal Transport, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LUg7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors study how to turn the single response-level ORM into token-level rewards for RLHF, without training an additional reward model. They propose Single-Source Entropic Optimal Transport (SEOT), which treats the final hidden state of the ORM as a single source and the hidden states of the response tokens as targets, with the transport cost given by their cosine similarity. The entropy-regularized problem has a closed-form solution and the resulting weights split the original reward across tokens while keeping its total unchanged. Experiments on mathematical reasoning and general-domain instruction following compare SEOT with training-based token-level reward models and other redistribution methods.

Critical strengths

The method is simple and cheap to use. SEOT needs only one forward pass of a frozen ORM, with no extra training or iterative optimization, and since the weights sum to one it leaves the original reward untouched. Across both backbones and the general-domain setting, SEOT improves over the other training-free redistribution baselines in Pass@1.

Critical weaknesses
  1. The paper is not yet sufficiently polished and feels somewhat repetitive. The same statement, "the outcome reward is predicted from its terminal hidden representation, while intermediate token representations exhibit different degrees of alignment with this terminal state", is restated almost word for word in the abstract, the introduction, and Section 4.1, making the writing feel monotonous. Yet the paper does not convincingly explain why this reward redistribution leads to better policy optimization.

  2. The hidden state space of the ORM has no specified geometry, and I could not find a reason why a larger cosine similarity to the final hidden state should mean a larger contribution to the preference judgment. The reward depends on the hidden states only through Wr⊤hm+TW_r^\top h_{m+T}.

  3. The hypothesis on the hidden states is not supported by a direct experiment. For example, if the critical token of a ground truth answer is replaced with a wrong one, the cosine similarity of that token to the final hidden state should change noticeably, while descriptive tokens should be largely unaffected.

  4. In Figures 4~6, critical tokens do not receive noticeably higher weights than the other tokens. The weights stay within a narrow range, which reads as the reward being spread almost uniformly by the softmax. A uniform redistribution baseline is missing, so it is unclear whether the gains come from the hidden state similarity or simply from spreading the reward evenly. This is my main concern, and I would be willing to raise my score depending on the authors' explanation of this point.

Minor issues

  • Line 344: BT-ORM is not defined.
  • τ\tau denotes both the trajectory and the entropy coefficient (e.g., Eq. 5 and Algorithm 1).
  • Figure 4 in Appendix D would help the reader if moved to the main text.
  • Line 352: "three types of reward distribution methods" →\rightarrow "three types of reward redistribution methods"
Questions for authors
  1. Could the authors report results with uniform redistribution under the same GAE-free PPO setting?

  2. When the critical token of a ground truth answer is replaced with an incorrect one, how much does its cosine similarity to the final hidden state change compared with that of descriptive tokens?

Reviewer_CQatOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SEOT, a training-free way to redistribute an outcome reward model’s (ORM) scalar reward across response tokens for PPO-based RLHF. Each token’s weight is a softmax, over the response, of the cosine similarity between that token’s ORM hidden state and the ORM hidden state at the final position. The authors motivate this softmax as the closed-form solution of a single-source entropic optimal transport problem. On GSM8K/MATH with 2–3B policies, and on a general-domain setup with Skywork-Reward, SEOT improves over PPO+ORM and the training-free baselines ABC and RED (e.g., +3.2 Pass@1 on both GSM8K and MATH for Llama-3.2-3B). It also trains more stably than these alternatives, though QRM remains stronger on Qwen3.5-2B.

Critical strengths
  1. Simplicity and cost. SEOT needs one forward pass of the frozen ORM, has a closed form and a single hyperparameter, and leaves the ORM untouched. If its gains survive the controls requested below, it is an easy drop-in for existing pipelines.

  2. A useful stability observation. Under GAE-free PPO, SEOT trains stably on MATH. In contrast, DPO-RM, QRM, ABC and RED degrade, and ABC drops to near-zero Pass@1 at one point (Fig. 3). Even if the mechanism differs from the one claimed (see W1), the observation that a broad allocation of the outcome reward stabilizes this optimizer is practically relevant.

Critical weaknesses

W1. Similarity is not credit, and SEOT is effectively near-uniform redistribution.

A token’s credit is counterfactual: how much choosing it, rather than an alternative, changes the expected outcome. Credit can be positive or negative. Cosine similarity to the terminal hidden state has none of these properties:

The weights are non-negative, so SEOT cannot penalize the erroneous step in an otherwise correct-looking response. The similarity ignores the reward head and uses hidden states that were never trained for this purpose. The final token always receives the largest weight, because its similarity to itself is 1.

Empirically, the weights are close to uniform: the top 10% of tokens receive 11.72% of the weight (Fig. 2b). The optimal-transport framing reduces to a plain softmax and adds no structure. The evidence for preference-awareness (Fig. 2a) has no uniform or position baseline. As a result, the gains cannot be attributed to the proposed mechanism without a uniform-redistribution control and a sweep over τ\tau.

W2. The comparison favors broad allocations and omits key baselines.

Handicapped main baseline. PPO+ORM uses GAE with γ=0.9\gamma=0.9, so the terminal reward barely reaches early tokens; PPO+ORM accordingly performs at the level of SFT.

Protocol biased toward flat allocations. The GAE-free protocol trains each token only on its own reward, which structurally penalizes concentrated or exploitable token rewards.

Missing critic-free baselines. GRPO and RLOO with the same ORM are missing, although QRM, whose setup the paper follows, showed both outperforming PPO+ORM. With near-uniform weights, SEOT is essentially a sequence-level broadcast scaled by 1/T1/T, so this comparison is decisive.

Unexplained checkpoint selection. In Table 1, the baselines on MATH (≈26) match their initial values in Fig. 3 rather than their degraded final values, and the checkpoint selection is not described.

Thin preference-alignment evidence. The only preference benchmark is AlpacaEval 2.0, reported as raw win rate with a non-standard judge. The gain (+0.74) is within noise.

Questions for authors
  1. Under the same PPO setup, how do uniform redistribution (rt=R/Tr_t=R/T), a τ\tau sweep, GRPO/RLOO with the same ORM, and PPO+ORM with γ=1\gamma=1 perform? This would most affect my assessment.
  2. Which checkpoint is reported in Tables 1–2, and how was it selected?
  3. Is ABC’s near-zero Pass@1 in Fig. 3 a formatting or termination failure rather than degraded reasoning?
  4. Why are rewards and value predictions standardized with separate batch statistics in Alg. 2?
Reviewer_z5cLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SEOT, which redistributes a frozen outcome reward model's response-level score using softmax weights derived from cosine similarities between response-token hidden states and the terminal hidden state. The entropy-regularized formulation admits a closed-form solution and preserves the sum of the redistributed rewards without additional reward-model training.

Critical strengths
  1. The redistribution mechanism is simple and practical to implement. SEOT obtains the response-level score and token representations from a single forward pass through a frozen ORM, with no additional reward-model training.

  2. The reported results show benefits across multiple evaluation settings. SEOT improves Pass@1 over ORM, ABC, and RED on both mathematical-reasoning benchmarks, and the general-domain experiment reports gains on AlpacaEval 2.0, MATH-500, and RACE.

Critical weaknesses
  1. The OT formulation offers limited novelty, and the preference-aware interpretation needs stronger support. With a single source and freely optimized target masses, the formulation reduces to entropy-regularized optimization over the probability simplex, yielding a softmax over hidden-state similarities. The derivation is valid, but I find the additional methodological contribution of the OT interpretation limited beyond providing a mathematical formulation of softmax token weighting. The substantive design choice is the similarity to the terminal ORM representation. It remains insufficiently established that this similarity measures a token's contribution to the final preference judgment, rather than representational alignment alone.

  2. The evaluation does not sufficiently isolate the source of the gains. A key missing comparison is uniform redistribution of the same response-level reward under otherwise identical optimization and normalization settings. This is especially relevant because the paper reports that the top 10% of tokens receive only 11.72% of the total weight, which is close to the 10% expected under uniform allocation. Without this baseline, it is difficult to determine whether the gains come from preference-aware weighting or primarily from distributing rewards more smoothly. I also find the experimental coverage insufficiently representative of current RLHF practice for an ICLR 2027 submission. Although Qwen3.5-2B is included, the policy evaluation is limited to two small-model backbones, with comparisons centered on ORM, QRM, DPO-RM, ABC, and RED and benchmarks such as GSM8K, MATH, AlpacaEval 2.0, and RACE. These experiments provide useful initial evidence, but leave the method's value in more recent training and evaluation settings unclear.

Questions for authors
  1. What evidence supports using similarity to the terminal ORM hidden state as a measure of token importance for the preference judgment?
  2. How does SEOT compare with uniform reward redistribution under the same PPO, advantage-estimation, reward-normalization, and tuning settings? Comparisons with shuffled similarity weights would also help determine whether the gains depend on preference-aware allocation or mainly on smoother reward distribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.