- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CORE, a reinforcement learning framework designed to improve contextual faithfulness in LLMs. The authors argue that existing outcome-based RL methods mainly reward final-answer correctness without explicitly encouraging models to use the provided context, which can lead models to rely on their parametric knowledge instead. To address this, CORE introduces a contextual influence score based on the difference in token log-probabilities with and without the provided context. This signal is used as an additional reward during policy optimization. To prevent reward hacking through excessive context copying, the authors introduce an accuracy-gating mechanism that applies the contextual influence reward only when the generated answer is correct. Experiments across four model configurations, two RL algorithms (PPO and GRPO), and several IID and OOD benchmarks show improvements in both task accuracy and LLM-judged contextual faithfulness.
Critical strengths
- The paper addresses an important problem in context-grounded generation, where models may rely on parametric knowledge instead of the provided context. 2.The contextual influence score is straightforward to compute and can be integrated into existing RL algorithms without modifying their optimization objectives.
- The experiments cover multiple model scales, architectures, and both PPO and GRPO, with consistent improvements over the corresponding baselines.
Critical weaknesses
-
Core assumption of CORE is that a larger difference in token likelihood with and without context indicates stronger contextual faithfulness. But the score in Equation 4 measures how much the context influences the generation of a token, it does not distinguish whether it comes from relevant evidence or simply from copying information in the context? For example, suppose the context states a fact A and also contains an unrelated reference number B. The question only asks a factual question regarding A. A response that copies B in its reasoning before giving the correct answer could receive additional contextual influence reward, even though B does not help answer the question about A. The accuracy gate does not necessarily prevent this, it only checks whether the final answer is correct. I think a more direct analysis of such potential reward hacking problem would be very helpful.
-
The main claim of the paper is that CORE provides fine-grained process supervision through token-level contextual influence. But according to Equations 6 and 7, the individual token scores are averaged into a single scalar reward for the entire trajectory. This reward is used for standard PPO or GRPO optimization. This seems to weaken the motivation for introducing token-level contextual influence. Although the reward is computed from token-level statistics, the optimization does not appear to directly provide dense token-level credit assignment.
-
Most results in Table 1 measure final-answer accuracy, which does not directly reveal whether the model relied on the provided context or its parametric knowledge. The paper includes an LLM-as-a-Judge evaluation, but the judging criterion mainly checks whether the response contradicts the context. A response can be consistent with the context without actually using it, when a model's parametric knowledge agrees with the provided context, it could produce a correct and non-contradictory answer without relying on the context at all.
-
The comparison with existing baselines is insufficient, only with PPO and GRPO and CAD. Several existing methods explicitly aim to improve contextual faithfulness through training, including Context-DPO (Bi et al., 2025), CANOE/Dual-GRPO (Si et al., 2026), RHIO (Huang et al., 2025), REFO (Wang et al., 2026), and KAFT (Li et al., 2023). Context-DPO and CANOE are already cited in the paper but are not included in the experimental comparison. These methods seem more relevant for evaluating the effectiveness of CORE than standard outcome-based RL baselines. I think comparisons with these, or a clear justification for their omission, would help better establish the empirical contribution of CORE. It would also be useful to include simpler reward-design baselines, such as using contextual likelihood without the context-free contrast or applying the contrastive reward only to final-answer tokens, to better isolate the benefits of the proposed reward formulation.
Questions for authors
See the weaknesses.