acceptodds
Under review as a conference paper at ICLR 2027

CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models

Abstract

With the growing use of Retrieval-Augmented Generation (RAG), improving contextual faithfulness and retrieval-grounded reasoning in large language models (LLMs) has become increasingly important. Existing RAG-oriented reinforcement learning (RL) methods primarily rely on external reward signals, such as correctness or LLM-based judges, which often provide indirect or noisy supervision for document faithfulness. Meanwhile, internal self-reward signals based on model confidence lack explicit grounding in retrieved evidence and may reinforce spurious behaviors during training. To address these limitations, we propose CTRL-RAG, an internal–external hybrid reward framework centered on a Contrastive Likelihood Reward (CLR). CLR measures the log-likelihood difference between responses generated with and without supporting evidence, directly estimating the contribution of retrieved documents to generation. This encourages the model to identify and utilize relevant evidence while suppressing reliance on noisy context. Combined with a correctness reward through a gating mechanism, CTRL-RAG encourages the model to use relevant evidence while treating correctness as a prerequisite. Experiments on single-hop, multi-hop, biomedical, and faithfulness benchmarks show consistent improvements in retrieval-grounded reasoning and contextual faithfulness across both dense and Mixture-of-Experts architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Faithfulness, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_euBzOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper adopts an internal-external hybrid reward framework centered on Contrastive Likelihood Reward (CLR) to train the model to assess document reliability during the RAG process. Experiments conducted across multiple base models as well as single-hop question answering, multi-hop question answering, biomedical and faithfulness benchmark datasets demonstrate that the training is effective. However, this paper contains several pieces of unreasonable experimental evidence, such as inconsistent tables and training curves that do not correspond to the method.

Critical strengths
  • This paper does not merely focus on the model's response output; instead, it treats the difference in responses before and after removing the context as the learning reward. This idea is intuitive and reasonable and has certain exploratory value.

  • In addition, the study conducts extensive experiments covering various models such as dense models and Mixture-of-Experts models, as well as multiple datasets, with a broad evaluation scope.

Critical weaknesses
  • The reward scale in Figure 5 is inconsistent with the method definition, and the training trajectories lack sufficient explanation. According to Equations (12)–(13), CLR is normalized via within-group min-max normalization and then multiplied by the binary correct reward, so the value range of the resulting hybrid reward should be [0,1]. However, the curve labeled "Hybrid CLR" in Figure 5 clearly exceeds this interval.

  • In addition, CLR only normalizes over T\sqrt T and is still affected by TT. When the average contribution of each token remains constant, the reward still rises as T\sqrt T increases. Therefore, an increase in reward cannot be directly interpreted as an improvement in evidence utilization capability. At this point, reward growth alone cannot measure capability improvement. Nevertheless, it is strange that the reward keeps rising in Figure 5. How can this happen if the sequence lengths are inconsistent? Alternatively, there may be an issue of non-unique influencing variables.

  • Moreover, the local variation trends of the CLR curve and the hybrid reward curve are surprisingly highly similar in the early stage, with even identical fluctuations. I am curious about why this occurs. If these two rewards only change linearly, why do their trends diverge later?

Questions for authors

It is unclear whether smoothing or aggregation methods were used for Figure 5. If so, we hope the authors clearly elaborate on this in the manuscript, as well as why the different reward curves are so smooth and consistent.

9 comments

Response to Reviewer

Comment 1: Reward scale inconsistency in Figure 5

The reward scale in Figure 5 appears inconsistent with the method definition. According to Equations (12)–(13), the Hybrid CLR reward should lie within [0,1][0,1], whereas the plotted curve exceeds this range.

Response

We apologize for the lack of clarification in the original manuscript. For visualization only, we rescaled the Hybrid CLR curve in Figure 5 so that it could be displayed together with the CLR curve on the same axis. Thus, the plotted values do not correspond to the original Hybrid CLR reward, whose range remains [0,1][0,1] as defined in our method. This rescaling is also implemented in the released code. We will explicitly describe this plotting procedure in the revised manuscript.


Comment 2: Length normalization and interpretation of reward growth

The reviewer points out that normalization by T\sqrt{T} does not completely eliminate the effect of response length. Therefore, reward growth alone cannot be interpreted as improved evidence utilization. The reviewer also asks why the reward continues to increase despite variations in response length.

Response

Thank you for raising this important concern. We agree that normalization by T\sqrt{T} does not completely eliminate the effect of response length on the CLR reward. We clarify below our motivation for this design choice and how the reward growth should be interpreted.

1. Why do we use T\sqrt{T} instead of TT for length normalization?

Our goal is not to prevent response length from increasing. During GRPO training, response length may naturally grow alongside model capability. Instead, we aim to mitigate length-driven reward hacking while retaining incentives for generating sufficiently informative responses.

The CLR reward accumulates token-level evidential contributions:

E(y)=∑t=1Tϵ(yt)E(y) = \sum_{t=1}^{T} \epsilon(y_t)

Without length normalization, the model could obtain higher rewards simply by generating more tokens, even if their individual evidential contributions are small. To address this issue, we consider two normalization strategies:

RT(y)=E(y)T=1T∑t=1Tϵ(yt)R_T(y) = \frac{E(y)}{T} = \frac{1}{T}\sum_{t=1}^{T}\epsilon(y_t)

and

RCLR(y)=E(y)TR_{\mathrm{CLR}}(y) = \frac{E(y)}{\sqrt{T}}

The key difference between these two strategies lies in the degree of penalty imposed on additional tokens, rather than whether they completely eliminate the effect of response length. Compared with TT-normalization, T\sqrt{T}-normalization imposes a weaker, sublinear penalty on length. This better balances discouraging reward hacking and preserving incentives for sufficiently informative responses.

When the threshold condition is satisfied, we can rewrite CLR as:

RCLR(y)=ϵˉ(y)T,ϵˉ(y)=E(y)TR_{\mathrm{CLR}}(y) = \bar{\epsilon}(y)\sqrt{T}, \quad \bar{\epsilon}(y) = \frac{E(y)}{T}

We acknowledge that the reward increases with T\sqrt{T} if the average token-level contribution remains constant. However, maintaining a constant average contribution as the response grows is not trivial. As shown in Figure 7, tokens repeating previously generated content generally have low evidential contributions. Thus, simply adding repetitive tokens is unlikely to maintain the average contribution. Longer responses need to provide additional evidential value to sustain or improve their average contribution.

More formally, suppose an additional token has evidential contribution ϵnew\epsilon_{\mathrm{new}}. Assuming the threshold condition remains satisfied, the reward increases only if

E+ϵnewT+1>ET\frac{E+\epsilon_{\mathrm{new}}}{\sqrt{T+1}} > \frac{E}{\sqrt{T}}

or equivalently,

ϵnew>E(T+1T−1)\epsilon_{\mathrm{new}} > E\left(\sqrt{\frac{T+1}{T}} - 1\right)

This shows that additional tokens must provide sufficient evidential contribution to compensate for the increased length penalty.

2. How should we interpret the increasing reward?

We agree that reward growth alone does not directly demonstrate improved evidence utilization capability. To examine whether the reward increase is driven merely by response length, we provide two observations.

As shown in Figure 3, response length eventually stabilizes under T\sqrt{T}-normalization, without substantial further growth in the later training stages. However, the reward in Figure 5 continues to increase. This suggests that the reward improvement cannot be explained solely by increasing response length.

Nevertheless, we acknowledge that these observations do not establish that every reward increase corresponds to improved evidence utilization. CLR is influenced by both response length and token-level evidential contributions, which may also be affected by changes in the generated content distribution. We therefore view CLR as an optimization signal rather than a standalone measure of evidence utilization capability. Actual capability improvements should be assessed through independent downstream evaluations, such as the RAGQA and PRGB results reported in our paper.

AuthorsOpenReview ↗

Clarifications on Reward Trend Similarity and Curve Smoothness

1. Similar trends between CLR and Hybrid CLR in the early stage

The Hybrid reward is not a linear transformation of CLR. Instead, it applies an accuracy-based gate to the normalized CLR reward:

RHybrid=RCLR′⋅RaccR_{\text{Hybrid}} = R_{\text{CLR}}' \cdot R_{\text{acc}}

where

Racc∈{0,1}R_{\text{acc}} \in \{0, 1\}

indicates whether the generated response is correct.

In the early stage of training, the accuracy gate has only a limited effect on the overall reward dynamics. As a result, the Hybrid reward follows trends and local fluctuations similar to those of CLR. As training progresses, however, the correctness signal becomes increasingly influential, so the accuracy gate affects the reward distribution more strongly, causing the two curves to diverge.

2. Why the CLR reward curve is relatively smooth

First, we clarify that no smoothing operation was applied to the reward curves in Figure 5. We have also released our code to facilitate verification.

We attribute the relatively smooth CLR trajectory to two main factors: the continuity of its underlying likelihood-based signal with respect to model parameters, and the stability introduced by large-sample averaging. In this respect, our reward differs from typical discrete RL rewards, which often change only when the final outcome category changes and can therefore appear much more abrupt.

Specifically, CLR is constructed from the difference between two SFT-style token-level negative log-likelihoods:

\begin{cases} \sqrt{T}\left(L_{\text{ablated}} - L_{\text{full}}\right), & \text{if } E > \tau \\ 0, & \text{otherwise} \end{cases}$$ where $$L_{\text{full}} = -\frac{1}{T}\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, q, D)$$ and $$L_{\text{ablated}} = -\frac{1}{T}\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, q, D \backslash \{d^*\})$$ For comparison, the SFT objective is also based on token-level negative log-likelihood: $$L_{\text{SFT}}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, x)$$ Like SFT, CLR is ultimately determined by the model’s conditional token probabilities, which depend on the model parameters. Under a fixed input, generated response, and context configuration, the corresponding log-likelihood terms are continuous functions of the model parameters. Therefore, when parameter updates between consecutive training steps are not large, the induced changes in token probabilities, and thus in the CLR reward, are also typically limited. In this sense, the reward signal tends to evolve smoothly as the model parameters change gradually during training. In addition, each reward value in Figure 5 is estimated using a large batch of $128 \times 8 = 1024$ samples. Averaging over such a large number of rollouts reduces the variance introduced by stochastic generation and makes the observed trajectory more stable. We note that CLR is not globally continuous, since the thresholding operation and leave-one-out selection can still introduce occasional discontinuities. Nevertheless, in practice, the combination of gradual parameter updates and large-sample averaging makes the reward curves appear relatively smooth.

Thanks for the comments, but the response so far has failed to convince me. For example, plotting value changes of two different magnitudes on the same figure is unreasonable. The authors could have used a separate plot with a different scale for this. Regarding the reward curve and the interpretation of length T, I hope the authors can provide more experimental results or log instead of only offering explanations.

Reviewer_YZFPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CTRL-RAG, a reinforcement learning framework designed to improve contextual faithfulness in retrieval-augmented generation (RAG). The core contribution is a Contrastive Likelihood Reward (CLR), which measures the difference in the log-likelihood of a generated response between the full retrieved context and an ablated context where supporting documents are removed. The authors further introduce length normalization and a thresholding mechanism to stabilize the reward, and combine CLR with an accuracy reward through multiplicative gating.

Critical strengths
  1. The paper addresses the important distinction between answer correctness and contextual faithfulness in RAG. The proposed likelihood-based reward is intuitive, easy to understand, and can be integrated into existing RL training frameworks without requiring an additional reward model.

  2. The authors conduct experiments across multiple benchmarks, model architectures, and reasoning modes. The improvements over conventional accuracy-based, citation-based, and LLM-as-a-Judge rewards are encouraging, and the ablation studies provide additional insights into the proposed design.

Critical weaknesses
  1. The core CLR formulation is relatively simple, essentially computing the log-likelihood difference between full and evidence-ablated contexts. The additional components, including length normalization, thresholding, and accuracy gating, are also straightforward. The paper lacks sufficient theoretical or methodological insights to establish why this simple formulation constitutes a substantial contribution beyond existing contrastive likelihood and counterfactual attribution approaches.

  2. CLR encourages stronger dependence on retrieved documents, potentially at the expense of the model's parametric knowledge. However, retrieved evidence can be incomplete, misleading, or incorrect, and effective RAG systems should adaptively balance external context and internal knowledge. As discussed in To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (Huang et al., ICLR 2025), blindly prioritizing external evidence is not always desirable. The authors should investigate whether CTRL-RAG remains effective under conflicting or unreliable retrieved contexts.

  3. Different reward methods use different reward-specific training sample selection strategies, making it difficult to isolate the effectiveness of CLR itself. More controlled comparisons would strengthen the experimental conclusions. Additionally, Figure 4 appears to use AI-generated visual elements and lacks the clarity and precision expected of a technical methodology diagram. I suggest replacing it with a more conventional scientific illustration.

Questions for authors

See Weakness

3 comments
AuthorsOpenReview ↗

Response to Weakness 1&2

We sincerely thank the reviewer for the constructive feedback and insightful comments. Below, we address your concerns point by point, supported by new controlled experiments.


Response to Weakness 1: On the Novelty and Insight of the CLR Formulation

We appreciate your observation regarding the mathematical simplicity of CLR. However, we respectfully argue that our primary contribution lies in bridging the gap between counterfactual attribution and Reinforcement Learning (RL) for RAG faithfulness—a paradigm shift from external heuristics to intrinsic dependency signals.

  • Limitation of Existing Extrinsic RL Signals: Current RL approaches for faithfulness predominantly rely on extrinsic rewards (e.g., citation accuracy, NLI models, or LLM-as-a-judge). While useful, these black-box metrics suffer from a fundamental blind spot: they cannot distinguish whether a correct answer is genuinely derived from the retrieved document or merely recalled from the model’s parametric memory.
  • Our Methodological Insight: Rather than proposing a complex new attribution formula, our key insight is repurposing the intrinsic log-likelihood difference (between full and evidence-ablated contexts) from a post-hoc interpretability tool into a direct, scalable optimization objective. By combining this intrinsic model signal (log-prob shift) with extrinsic supervision (pre-annotated golden documents and accuracy gating), CLR directly penalizes parametric memorization and forces the model to internalize context-reliance during RL training.
  • Simplicity as a Feature for RL: In RL for LLMs, overly complex reward functions often lead to optimization instability and reward hacking. The simplicity of CLR, paired with our straightforward length normalization and gating, ensures low computational overhead and high training stability, which we empirically validate as highly effective across diverse benchmarks.

Response to Weakness 2: Balancing External Context and Parametric Knowledge

Thank you for pointing out the important direction of situated faithfulness (Huang et al., ICLR 2025) and the issue of unreliable retrieved contexts.

  • Scope and Task Definition: As noted in our paper and Limitations section, CTRL-RAG specifically targets the core challenge of strict context faithfulness under the assumption that retrieved documents contain valid evidence.
  • Practical Value & System Modularity: In many high-stakes, real-world RAG deployments (e.g., legal, medical, or enterprise knowledge bases), system engineering favors a modular (decoupled) design: information relevance/reliability is handled upstream by advanced retrievers and rerankers, while the generator is strictly required to adhere to the provided context to ensure auditability and prevent hallucination. In such pipelines, prioritizing external evidence over parametric knowledge is a deliberate and necessary design choice. Optimizing pure faithfulness in isolation remains a foundational pillar for RAG systems.
  • Integration with Situated Faithfulness (Future Direction): We fully agree that handling conflicting or noisy contexts is crucial for open-domain scenarios. Notably, our CLR framework is orthogonal and adaptable to the paradigm in Huang et al. (2025). For instance, if noisy documents are labeled during training, the CLR sign could be inverted (penalizing reliance on distracted contexts) or gated by a context-reliability classifier. We will prominently cite Huang et al. (2025) and expand our discussion in the revised paper to highlight how CLR can be extended to situated faithfulness.

AuthorsOpenReview ↗

Response to Weakness 3

Response to Weakness 3: Isolating Reward Effects from Data Selection & Figure 4

Thank you for highlighting this concern. We agree that sharing an initialization and a raw candidate pool alone does not separate the effect of reward design from reward-specific data selection.

To address this confound, we conducted a controlled ablation in which Accuracy, CLR-only, and Hybrid use the same training prompts in the same order, the same SFT checkpoint, and identical response-sampling budgets and optimization schedules. We disable all reward-specific prompt selection, including CLR’s log-probability standard-deviation threshold. No groups are discarded or resampled based on reward variance.

Without reward-specific filtering, Accuracy, CLR-only, and Hybrid achieve Multi-hop scores of 74.70, 76.97, and 77.27, respectively, and PRGB scores of 74.2, 76.3, and 78.0. Relative to Accuracy, CLR-only and Hybrid improve Multi-hop by 2.27 and 2.57 percentage points, and PRGB by 2.1 and 3.8 points, respectively. These results support a benefit from CLR-based rewards beyond reward-specific data selection.

As a complementary check, applying CLR-based selection to Accuracy reduces its Multi-hop score from 74.70 to 72.50, below the SFT baseline (73.47). This indicates that CLR-based selection is not uniformly advantageous across rewards.

Reward-matched selection generally provides additional gains, particularly on Multi-hop, although not every metric improves. We will add this controlled ablation and clarify the selection criteria, distinguishing the reward comparison under shared training conditions from the original reward-and-selection pipeline comparison.

All results below use no-think evaluation. Multi-hop and Single-hop are the unweighted averages over 2Wiki/HotpotQA/MuSiQue and TriviaQA/PopQA, respectively. Values referenced in the response are shown in bold.

Reward / ModelSelectionMulti-hop2WikiHotpotQAMuSiQueSingle-hopTQAPQAPRGBPubMedQA
SFTBaseline73.4778.778.363.492.2592.292.373.864.2
AccuracyNo reward-specific filtering74.7079.779.465.092.9093.592.374.263.3
AccuracyCLR-based selection72.5077.678.261.790.8591.390.471.363.1
AccuracyAccuracy-based selection75.8380.580.866.292.8592.593.276.770.4
R_totalAccuracy-based selection77.4781.182.269.192.9593.492.578.371.6
CLR-onlyNo reward-specific filtering76.9781.181.168.794.8094.994.776.375.4
CLR-onlyCLR-based selection78.7783.083.769.693.3092.993.778.576.8
HybridNo reward-specific filtering77.2783.281.067.692.5593.092.178.076.8
HybridCLR-based selection79.2082.984.170.694.7094.894.679.577.9

2. Revision of Figure 4

We apologize that the visual style of Figure 4 detracted from the technical clarity of our paper. We have completely redrawn Figure 4 using standard scientific illustration tools to provide a clean, precise, and conventional methodological diagram for the revised manuscript.

I appreciate the authors’ response and their careful answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

Reviewer_RvoEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces CTRL-RAG, a reinforcement learning framework that rewards evidence use by comparing the likelihood of the same response under the full context and a context with supporting documents removed. The resulting contrastive likelihood reward is combined with an answer correctness reward. Experiments show improvements across several RAG benchmarks and two model architectures. However, the comparisons confound reward design with training data selection, and the connection between likelihood differences and semantic faithfulness remains insufficiently validated. Inconsistencies in the description of the reward further limit confidence in the paper’s conclusions.

Critical strengths
  1. The paper addresses an important distinction between producing a correct answer and producing an answer grounded in the supplied evidence. This is a relevant problem for improving the reliability of RAG systems.
  2. Comparing response likelihoods before and after removing supporting evidence is intuitive and provides a more fine-grained training signal than answer correctness alone.
  3. The experiments cover multi-hop, single-hop, biomedical QA, and a faithfulness benchmark, with results on both dense and MoE models. The paper also examines alternative evidence removal and reward combination strategies.
Critical weaknesses
  1. According to the main text and Appendix A.1, each reward method selects its own training subset, with CLR additionally applying a threshold on the standard deviation of log-probabilities. Sharing an initial checkpoint and a raw candidate pool therefore does not ensure comparable training data: the methods may differ in sample difficulty, dataset composition, and the amount of useful training signal. It is unclear how much of the improvement comes from CLR rather than data selection. The explanation concerning within-group reward variance is reasonable, but does not resolve this confound. Comparisons using the same prompt stream and sampling budget, or experiments crossing reward functions with selection strategies, are needed.

  2. A decrease in likelihood after evidence removal does not necessarily establish faithfulness. A response could copy entities, dates, or citation markers from supporting documents while making unsupported inferences, yet still receive a high contrastive score. Appendix B treats large contributions from citation markers as evidence of grounding, without ruling out this superficial dependence. Targeted tests involving incorrect reasoning, irrelevant copying, and fabricated citations would help establish what CLR actually measures. Controls for changes in context length and document position are also missing. As presented, CLR is better justified as a proxy for context sensitivity than as a direct measure of semantic faithfulness.

  3. In Figure 6, perplexity under the full context remains largely stable, while perplexity after removing supporting documents increases throughout training. The authors interpret this as stronger evidence reliance, but the same pattern could arise because the model becomes worse at producing the original response under incomplete evidence. It does not, by itself, demonstrate better evidence understanding. Figure 2 shows stable retrieval-free accuracy, but this does not resolve the issue: receiving no documents is different from receiving only distractors. Results on fixed held-out examples under full evidence, partial evidence, and distractor-only contexts are needed to distinguish these explanations.

  4. The explanation of reward truncation is inaccurate. Setting non-positive rewards to zero does not exclude those responses from GRPO updates: they can still receive negative advantages when other responses in the group have positive rewards. These samples therefore still influence optimization. There is also a discrepancy in the reported reward scale: Equations (12) and (13) imply that the hybrid reward lies between zero and one, whereas the “Hybrid CLR” curves in Figure 5 are well above this range. If the figure reports an unnormalized diagnostic score, it should say so. Otherwise, there is an unresolved discrepancy between the stated objective and the implementation.

  5. The ablations do not fully support the conclusions about length normalization and correctness gating. Figure 3 reports response length alone, which is insufficient to conclude that linear normalization prevents effective learning; corresponding accuracy, faithfulness, and repetition measurements are needed. The benefits of multiplicative gating are also inconsistent. In the think setting of Table 1, the hybrid reward has a slightly lower average score than CLR alone. In Table 2, multiplication underperforms addition for LOO-Avg and improves the Remove-All average by only 0.3 points. Without multiple seeds or confidence intervals, it is difficult to assess whether these small differences are reliable.

  6. CLR depends on supporting-document annotations, yet the paper provides limited detail on how rules and LLM judgments are combined, how accurate these annotations are, or how bridge documents that do not contain the answer are identified. Similar gaps affect the judge baselines. The main text describes sentence-level supervision, while the appendix specifies paragraph-level judgments. The judge prompt also rejects content requiring non-trivial inference, which may disadvantage valid multi-hop reasoning. The aggregation of paragraph labels, treatment of non-factual content, and combination of faithfulness with accuracy rewards should be specified. Without these details, I cannot tell whether the reported advantage partly reflects annotation quality or choices in the baseline implementation.

Questions for authors
  1. Abstract, lines 21–22: Clarify that the method compares the likelihood of the same response under two contexts, rather than comparing two separately generated responses.
  2. Tables 1–2: Some Hotpot/2Wiki and PQA/TQA entries appear to be swapped across the tables. Please check the column labels and underlying results.
  3. Appendix A.1: pass@8 appears to denote the fraction of correct responses among eight samples. This differs from the standard meaning of pass@k and should be clarified.
  4. Line 189: Replace “Overall Methodology Figure is shown in Figure 4” with “An overview of the method is shown in Figure 4.”
  5. Line 310 and Appendix B: Add the missing spaces in LLM(GPT-5.4) and Figure7.
  6. Appendix C: Several citations run into the preceding text, such as RagQALeaderboardZhehao... and PRGBTan.... Please fix the citation formatting.
  7. Appendix E: The text refers to “Prompt E.1/E.2,” while the boxes are labeled “Prompt 1/2.” Use consistent numbering.
8 comments
AuthorsOpenReview ↗

Response to Weakness 1: Separating Reward Effects from Data Selection

Response to Weakness 1: Separating Reward Effects from Data Selection

Thank you for highlighting this concern. We agree that sharing an initialization and a raw candidate pool alone does not separate the effect of reward design from reward-specific data selection.

To address this confound, we conducted a controlled ablation in which Accuracy, CLR-only, and Hybrid use the same training prompts in the same order, the same SFT checkpoint, and identical response-sampling budgets and optimization schedules. We disable all reward-specific prompt selection, including CLR’s log-probability standard-deviation threshold. No groups are discarded or resampled based on reward variance.

Without reward-specific filtering, Accuracy, CLR-only, and Hybrid achieve Multi-hop scores of 74.70, 76.97, and 77.27, respectively, and PRGB scores of 74.2, 76.3, and 78.0. Relative to Accuracy, CLR-only and Hybrid improve Multi-hop by 2.27 and 2.57 percentage points, and PRGB by 2.1 and 3.8 points, respectively. These results support a benefit from CLR-based rewards beyond reward-specific data selection.

As a complementary check, applying CLR-based selection to Accuracy reduces its Multi-hop score from 74.70 to 72.50, below the SFT baseline (73.47). This indicates that CLR-based selection is not uniformly advantageous across rewards.

Reward-matched selection generally provides additional gains, particularly on Multi-hop, although not every metric improves. We will add this controlled ablation and clarify the selection criteria, distinguishing the reward comparison under shared training conditions from the original reward-and-selection pipeline comparison.

All results below use no-think evaluation. Multi-hop and Single-hop are the unweighted averages over 2Wiki/HotpotQA/MuSiQue and TriviaQA/PopQA, respectively. Values referenced in the response are shown in bold.

Reward / ModelSelectionMulti-hop2WikiHotpotQAMuSiQueSingle-hopTQAPQAPRGBPubMedQA
SFTBaseline73.4778.778.363.492.2592.292.373.864.2
AccuracyNo reward-specific filtering74.7079.779.465.092.9093.592.374.263.3
AccuracyCLR-based selection72.5077.678.261.790.8591.390.471.363.1
AccuracyAccuracy-based selection75.8380.580.866.292.8592.593.276.770.4
R_totalAccuracy-based selection77.4781.182.269.192.9593.492.578.371.6
CLR-onlyNo reward-specific filtering76.9781.181.168.794.8094.994.776.375.4
CLR-onlyCLR-based selection78.7783.083.769.693.3092.993.778.576.8
HybridNo reward-specific filtering77.2783.281.067.692.5593.092.178.076.8
HybridCLR-based selection79.2082.984.170.694.7094.894.679.577.9
AuthorsOpenReview ↗

Response to Weakness 2

W2. Surface Copying and Semantic Faithfulness

We agree that CLR should be interpreted as a model-dependent proxy for evidence reliance. A high CLR alone does not guarantee that every inference or citation is semantically supported. To examine whether CLR distinguishes supported content beyond shared surface tokens, we conducted the following paired tests.

Scoring procedure. Each candidate response was held fixed and scored under both the full context and a context with the supporting document removed. We accumulated log-probability differences over answer tokens only. All measurements below use CLR before group normalization, without an accuracy gate. These are illustrative tests on the deployed model, rather than experiments on the paper’s checkpoint.

Profession replacement. We first compared supported and unsupported responses that share the prefix “Tom is a” but differ in the stated profession.

SettingSupported response: E / CLRUnsupported response: E / CLR
English: “Tom is a teacher/scientist.”20.222 / 9.044−1.973 / 0
English with citation “[1]”13.770 / 5.205−3.673 / 0

In the unsupported response, the shared prefix “Tom is a” contributes +13.184, whereas “scientist” contributes −15.292. Including the period, the sequence-level contribution is E = −1.973. Thus, in this example, the unsupported predicate offsets the positive contributions from the shared entity and phrasing.

The zero rewards are exact outputs of the τ = 0 truncation: R_CLR = max(E, 0) / √T. For the unsupported response, T = 5, giving approximately −0.882 before truncation and exactly 0 afterward. A second inference worker reproduced the supported-versus-unsupported ranking in both settings, with zero CLR for the unsupported responses.

Supported premise with an unsupported conclusion. We further tested a pair sharing the same correct premise, entity, syntactic frame, and citation, while changing the inferred degree.

Document: [1] Tom is a teacher at School A. All teachers at School A hold a master's degree in education.

Supported response: Tom is a teacher at School A, so he holds a master's degree in education.[1]

Unsupported response: Tom is a teacher at School A, so he holds a doctoral degree in education.[1]

The doctoral degree is not supported by the document, although it is not explicitly ruled out.

ConclusionAnswer tokens (T)ECLRConclusion contribution
Master's degree (supported)2028.8226.445+5.895
Doctoral degree (unsupported)1916.3313.747−5.643

Conclusion contributions exclude sentence-final punctuation and the citation.

The supported conclusion receives a positive contribution, whereas the unsupported conclusion receives a negative contribution. The supported response consequently receives the higher overall CLR, despite both candidates sharing the same correct premise and citation. A second inference worker reproduced this ranking, with CLR values of 5.441 versus 3.291.

The unsupported response nevertheless retains positive overall CLR because its correct premise and other tokens contribute positively to the sequence-level sum. This example therefore demonstrates relative discrimination between the two responses while also illustrating an important limitation: positive sequence-level CLR does not guarantee support for every individual inference.

We agree that high contributions from citation markers in Appendix B alone do not establish semantic faithfulness. We will present that visualization as qualitative evidence of model sensitivity to supporting documents, rather than proof that all generated claims are supported. Independent task and faithfulness evaluations support CLR’s usefulness as a training reward, but do not establish a general semantic-verification guarantee.

Evidence removal also changes context length and document positions, and our current analysis does not separately rule out these effects. Broader tests involving incorrect reasoning, irrelevant copying, and fabricated citations, together with matched-length context controls on the paper’s checkpoints, remain important for establishing the scope and reliability of this signal. We will make these limitations explicit in the revision.

AuthorsOpenReview ↗

Response to Weakness 3 and 4

W3. Interpreting Figure 6 and Evidence Dependence

Thank you for raising this alternative explanation. We agree that stable accuracy without documents does not establish stable performance with distractor-only contexts, and that Figure 6 alone cannot demonstrate improved evidence understanding.

To address this concern, we repeated the evaluation in Figure 2 using the same evaluation questions, model checkpoints, and generation and scoring settings. The only change was to remove all gold supporting documents while retaining the distractor documents.

Across the same checkpoints, from SFT to the final RL model, distractor-only accuracy fluctuates between 25% and 26%, with no sustained downward trend. Meanwhile, accuracy with full supporting evidence improves during training, as shown in Figure 2. Thus, we observe improved answering performance when supporting evidence is available, without a corresponding decline in answering accuracy under distractor-only contexts. This provides evidence against a general deterioration in distractor-only task performance as the explanation for the observed training trends.

Stable answer accuracy does not, by itself, establish stable likelihood for a particular fixed response. We therefore interpret Figure 6 as a diagnostic observation consistent with increased evidence dependence, rather than standalone proof of improved evidence understanding. We will clarify this distinction in the revision and base our interpretation jointly on the likelihood analysis, the full-evidence performance gains, the distractor-only control, and the independent task and faithfulness evaluations.

W4. Reward Truncation and the Scale in Figure 5

Non-positive contribution truncation. We agree that our original wording was inaccurate: assigning zero reward does not exclude a response from GRPO updates. When other responses in the group have positive rewards, zero-reward responses can still receive negative advantages and influence optimization.

The purpose of truncation is to remove distinctions based on potentially unreliable negative magnitudes. As discussed in §4.2, negative evidential contributions can arise from document noise, context-length changes, or conflicts with parametric knowledge; their magnitudes do not necessarily provide a reliable ordering of response faithfulness. We therefore assign a common reward of zero to responses with non-positive sequence-level contributions, while retaining their participation in within-group comparisons and GRPO updates.

Table 2 provides empirical support for this choice: truncation improves the Remove-All average from 82.2 to 82.6, and improves LOO-Min and LOO-Avg by 0.5 percentage points each. We will revise §4.2 and Appendix A.6 to clarify that truncation removes distinctions among non-positive contribution magnitudes, without excluding the corresponding responses from optimization.

Figure 5. The figure reports raw CLR scores from Eq. (11) under CLR-only and Hybrid training, used as a diagnostic of training dynamics. Here, “raw” means before within-group min-max normalization and correctness gating; length normalization and non-positive contribution truncation have already been applied. These scores are therefore not restricted to [0,1].

The corresponding logging is already present in the code submitted with the supplementary material. Specifically, grpo/swift/megatron/trainers/grpo_trainer.py preserves the pre-normalization values as raw_rewards_per_func and separately logs their mean under rewards/faithful_margin_token/raw_mean. This distinguishes the raw CLR diagnostic from the normalized and combined reward used for optimization.

We retain this diagnostic because the absolute magnitude of CLR is informative: it tracks changes in the length-normalized likelihood contrast between full and ablated evidence. Within-group min-max normalization removes this absolute scale, so the normalized reward alone cannot reveal these changes.

In Hybrid training, the reward passed to GRPO is instead the normalized and correctness-gated reward defined in Eq. (13), R_hybrid = R'_CLR × R_acc, which remains within [0,1].

We will relabel the curves as “Raw CLR — CLR-only training” and “Raw CLR — Hybrid training”, and revise the caption to state:

Mean raw CLR scores (Eq. 11) during CLR-only and Hybrid training, after length normalization and non-positive contribution truncation but before within-group min-max normalization and correctness gating. These scores monitor the absolute scale of CLR and are distinct from the final Hybrid reward used for optimization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.