- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper adopts an internal-external hybrid reward framework centered on Contrastive Likelihood Reward (CLR) to train the model to assess document reliability during the RAG process. Experiments conducted across multiple base models as well as single-hop question answering, multi-hop question answering, biomedical and faithfulness benchmark datasets demonstrate that the training is effective. However, this paper contains several pieces of unreasonable experimental evidence, such as inconsistent tables and training curves that do not correspond to the method.
Critical strengths
-
This paper does not merely focus on the model's response output; instead, it treats the difference in responses before and after removing the context as the learning reward. This idea is intuitive and reasonable and has certain exploratory value.
-
In addition, the study conducts extensive experiments covering various models such as dense models and Mixture-of-Experts models, as well as multiple datasets, with a broad evaluation scope.
Critical weaknesses
-
The reward scale in Figure 5 is inconsistent with the method definition, and the training trajectories lack sufficient explanation. According to Equations (12)–(13), CLR is normalized via within-group min-max normalization and then multiplied by the binary correct reward, so the value range of the resulting hybrid reward should be [0,1]. However, the curve labeled "Hybrid CLR" in Figure 5 clearly exceeds this interval.
-
In addition, CLR only normalizes over and is still affected by . When the average contribution of each token remains constant, the reward still rises as increases. Therefore, an increase in reward cannot be directly interpreted as an improvement in evidence utilization capability. At this point, reward growth alone cannot measure capability improvement. Nevertheless, it is strange that the reward keeps rising in Figure 5. How can this happen if the sequence lengths are inconsistent? Alternatively, there may be an issue of non-unique influencing variables.
-
Moreover, the local variation trends of the CLR curve and the hybrid reward curve are surprisingly highly similar in the early stage, with even identical fluctuations. I am curious about why this occurs. If these two rewards only change linearly, why do their trends diverge later?
Questions for authors
It is unclear whether smoothing or aggregation methods were used for Figure 5. If so, we hope the authors clearly elaborate on this in the manuscript, as well as why the different reward curves are so smooth and consistent.
9 comments
Clarifications on Reward Trend Similarity and Curve Smoothness
1. Similar trends between CLR and Hybrid CLR in the early stage
The Hybrid reward is not a linear transformation of CLR. Instead, it applies an accuracy-based gate to the normalized CLR reward:
where
indicates whether the generated response is correct.
In the early stage of training, the accuracy gate has only a limited effect on the overall reward dynamics. As a result, the Hybrid reward follows trends and local fluctuations similar to those of CLR. As training progresses, however, the correctness signal becomes increasingly influential, so the accuracy gate affects the reward distribution more strongly, causing the two curves to diverge.
2. Why the CLR reward curve is relatively smooth
First, we clarify that no smoothing operation was applied to the reward curves in Figure 5. We have also released our code to facilitate verification.
We attribute the relatively smooth CLR trajectory to two main factors: the continuity of its underlying likelihood-based signal with respect to model parameters, and the stability introduced by large-sample averaging. In this respect, our reward differs from typical discrete RL rewards, which often change only when the final outcome category changes and can therefore appear much more abrupt.
Specifically, CLR is constructed from the difference between two SFT-style token-level negative log-likelihoods:
\begin{cases} \sqrt{T}\left(L_{\text{ablated}} - L_{\text{full}}\right), & \text{if } E > \tau \\ 0, & \text{otherwise} \end{cases}$$ where $$L_{\text{full}} = -\frac{1}{T}\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, q, D)$$ and $$L_{\text{ablated}} = -\frac{1}{T}\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, q, D \backslash \{d^*\})$$ For comparison, the SFT objective is also based on token-level negative log-likelihood: $$L_{\text{SFT}}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, x)$$ Like SFT, CLR is ultimately determined by the model’s conditional token probabilities, which depend on the model parameters. Under a fixed input, generated response, and context configuration, the corresponding log-likelihood terms are continuous functions of the model parameters. Therefore, when parameter updates between consecutive training steps are not large, the induced changes in token probabilities, and thus in the CLR reward, are also typically limited. In this sense, the reward signal tends to evolve smoothly as the model parameters change gradually during training. In addition, each reward value in Figure 5 is estimated using a large batch of $128 \times 8 = 1024$ samples. Averaging over such a large number of rollouts reduces the variance introduced by stochastic generation and makes the observed trajectory more stable. We note that CLR is not globally continuous, since the thresholding operation and leave-one-out selection can still introduce occasional discontinuities. Nevertheless, in practice, the combination of gradual parameter updates and large-sample averaging makes the reward curves appear relatively smooth.Thanks for the comments, but the response so far has failed to convince me. For example, plotting value changes of two different magnitudes on the same figure is unreasonable. The authors could have used a separate plot with a different scale for this. Regarding the reward curve and the interpretation of length T, I hope the authors can provide more experimental results or log instead of only offering explanations.
Response to Reviewer
Comment 1: Reward scale inconsistency in Figure 5
The reward scale in Figure 5 appears inconsistent with the method definition. According to Equations (12)–(13), the Hybrid CLR reward should lie within [0,1], whereas the plotted curve exceeds this range.
Response
We apologize for the lack of clarification in the original manuscript. For visualization only, we rescaled the Hybrid CLR curve in Figure 5 so that it could be displayed together with the CLR curve on the same axis. Thus, the plotted values do not correspond to the original Hybrid CLR reward, whose range remains [0,1] as defined in our method. This rescaling is also implemented in the released code. We will explicitly describe this plotting procedure in the revised manuscript.
Comment 2: Length normalization and interpretation of reward growth
The reviewer points out that normalization by T does not completely eliminate the effect of response length. Therefore, reward growth alone cannot be interpreted as improved evidence utilization. The reviewer also asks why the reward continues to increase despite variations in response length.
Response
Thank you for raising this important concern. We agree that normalization by T does not completely eliminate the effect of response length on the CLR reward. We clarify below our motivation for this design choice and how the reward growth should be interpreted.
1. Why do we use T instead of T for length normalization?
Our goal is not to prevent response length from increasing. During GRPO training, response length may naturally grow alongside model capability. Instead, we aim to mitigate length-driven reward hacking while retaining incentives for generating sufficiently informative responses.
The CLR reward accumulates token-level evidential contributions:
E(y)=∑t=1Tϵ(yt)
Without length normalization, the model could obtain higher rewards simply by generating more tokens, even if their individual evidential contributions are small. To address this issue, we consider two normalization strategies:
RT(y)=TE(y)=T1∑t=1Tϵ(yt)
and
RCLR(y)=TE(y)
The key difference between these two strategies lies in the degree of penalty imposed on additional tokens, rather than whether they completely eliminate the effect of response length. Compared with T-normalization, T-normalization imposes a weaker, sublinear penalty on length. This better balances discouraging reward hacking and preserving incentives for sufficiently informative responses.
When the threshold condition is satisfied, we can rewrite CLR as:
RCLR(y)=ϵˉ(y)T,ϵˉ(y)=TE(y)
We acknowledge that the reward increases with T if the average token-level contribution remains constant. However, maintaining a constant average contribution as the response grows is not trivial. As shown in Figure 7, tokens repeating previously generated content generally have low evidential contributions. Thus, simply adding repetitive tokens is unlikely to maintain the average contribution. Longer responses need to provide additional evidential value to sustain or improve their average contribution.
More formally, suppose an additional token has evidential contribution ϵnew. Assuming the threshold condition remains satisfied, the reward increases only if
T+1E+ϵnew>TE
or equivalently,
ϵnew>E(TT+1−1)
This shows that additional tokens must provide sufficient evidential contribution to compensate for the increased length penalty.
2. How should we interpret the increasing reward?
We agree that reward growth alone does not directly demonstrate improved evidence utilization capability. To examine whether the reward increase is driven merely by response length, we provide two observations.
As shown in Figure 3, response length eventually stabilizes under T-normalization, without substantial further growth in the later training stages. However, the reward in Figure 5 continues to increase. This suggests that the reward improvement cannot be explained solely by increasing response length.
Nevertheless, we acknowledge that these observations do not establish that every reward increase corresponds to improved evidence utilization. CLR is influenced by both response length and token-level evidential contributions, which may also be affected by changes in the generated content distribution. We therefore view CLR as an optimization signal rather than a standalone measure of evidence utilization capability. Actual capability improvements should be assessed through independent downstream evaluations, such as the RAGQA and PRGB results reported in our paper.