acceptodds
Withdrawn from ICLR 2027

CORE: Internalizing Context Faithfulness via Intervention-Derived Process Rewards in Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In context-grounded tasks, Large Language Models may underutilize provided contextual information and instead rely more heavily on their parametric knowledge. Although reinforcement learning can optimize models toward task-specific objectives, outcome-based rewards provide only sparse supervision and offer limited direct guidance on how the context should be used during generation. Process-based rewards provide finer-grained supervision over intermediate generation steps, but often rely on costly intermediate annotations. We propose CORE, a reinforcement learning framework for improving contextual faithfulness during generation. Given a trajectory generated with access to the context, CORE re-evaluates the likelihood of the same generated tokens after intervening on context availability and uses the resulting token-level likelihood changes to measure contextual influence. This signal is incorporated as a dense, annotation-free process reward during policy optimization. To mitigate reward exploitation through excessive context copying, CORE further introduces an accuracy-gating mechanism that applies the contextual influence reward only to correct trajectories, coupling contextual reliance with task correctness. Across multiple model architectures, reinforcement learning algorithms, and both in-distribution and out-of-distribution benchmarks, CORE consistently improves task accuracy over the corresponding baselines. Complementary LLM-as-a-Judge evaluations further indicate improved contextual faithfulness on the evaluated settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Reinforcement Learning, Context Faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P8yLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CORE, a reinforcement learning framework designed to improve contextual faithfulness in LLMs. The authors argue that existing outcome-based RL methods mainly reward final-answer correctness without explicitly encouraging models to use the provided context, which can lead models to rely on their parametric knowledge instead. To address this, CORE introduces a contextual influence score based on the difference in token log-probabilities with and without the provided context. This signal is used as an additional reward during policy optimization. To prevent reward hacking through excessive context copying, the authors introduce an accuracy-gating mechanism that applies the contextual influence reward only when the generated answer is correct. Experiments across four model configurations, two RL algorithms (PPO and GRPO), and several IID and OOD benchmarks show improvements in both task accuracy and LLM-judged contextual faithfulness.

Critical strengths
  1. The paper addresses an important problem in context-grounded generation, where models may rely on parametric knowledge instead of the provided context. 2.The contextual influence score is straightforward to compute and can be integrated into existing RL algorithms without modifying their optimization objectives.
  2. The experiments cover multiple model scales, architectures, and both PPO and GRPO, with consistent improvements over the corresponding baselines.
Critical weaknesses
  1. Core assumption of CORE is that a larger difference in token likelihood with and without context indicates stronger contextual faithfulness. But the score in Equation 4 measures how much the context influences the generation of a token, it does not distinguish whether it comes from relevant evidence or simply from copying information in the context? For example, suppose the context states a fact A and also contains an unrelated reference number B. The question only asks a factual question regarding A. A response that copies B in its reasoning before giving the correct answer could receive additional contextual influence reward, even though B does not help answer the question about A. The accuracy gate does not necessarily prevent this, it only checks whether the final answer is correct. I think a more direct analysis of such potential reward hacking problem would be very helpful.

  2. The main claim of the paper is that CORE provides fine-grained process supervision through token-level contextual influence. But according to Equations 6 and 7, the individual token scores are averaged into a single scalar reward for the entire trajectory. This reward is used for standard PPO or GRPO optimization. This seems to weaken the motivation for introducing token-level contextual influence. Although the reward is computed from token-level statistics, the optimization does not appear to directly provide dense token-level credit assignment.

  3. Most results in Table 1 measure final-answer accuracy, which does not directly reveal whether the model relied on the provided context or its parametric knowledge. The paper includes an LLM-as-a-Judge evaluation, but the judging criterion mainly checks whether the response contradicts the context. A response can be consistent with the context without actually using it, when a model's parametric knowledge agrees with the provided context, it could produce a correct and non-contradictory answer without relying on the context at all.

  4. The comparison with existing baselines is insufficient, only with PPO and GRPO and CAD. Several existing methods explicitly aim to improve contextual faithfulness through training, including Context-DPO (Bi et al., 2025), CANOE/Dual-GRPO (Si et al., 2026), RHIO (Huang et al., 2025), REFO (Wang et al., 2026), and KAFT (Li et al., 2023). Context-DPO and CANOE are already cited in the paper but are not included in the experimental comparison. These methods seem more relevant for evaluating the effectiveness of CORE than standard outcome-based RL baselines. I think comparisons with these, or a clear justification for their omission, would help better establish the empirical contribution of CORE. It would also be useful to include simpler reward-design baselines, such as using contextual likelihood without the context-free contrast or applying the contrastive reward only to final-answer tokens, to better isolate the benefits of the proposed reward formulation.

Questions for authors

See the weaknesses.

Reviewer_VbmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the CORE method to enhance the context fidelity of LLMs during reinforcement learning fine-tuning. This method measures the influence of the context through the difference in token log probabilities between context-conditioned input and context-masked input. These token-level scores are averaged over the generated trajectories and combined with task accuracy rewards through an accuracy gating mechanism.

Critical strengths
  1. A simple and well-motivated reward formulation.
  2. Consistent empirical improvements with useful ablation evidence.
Critical weaknesses
  1. The difference between token-level contextual influence and process-level reinforcement is not yet clear.
  2. This assessment fails to fully demonstrate its superiority over existing context-fidelity training methods.
  3. Figure 1 does not clearly illustrate the core mechanism of CORE.
Questions for authors

Q1. Token-level process rewards.

Formula (4) calculates the token-level context influence score, but formula (6) averages it to form a trajectory-level reward. In GRPO, this scalar reward is used to calculate the shared advantage of all tokens in the trajectory. Therefore, although the paper emphasizes process-level supervision, the optimization process does not directly provide specific credit allocation for each token.

Has the author considered using a single token-level influence score to construct a token-level reward or advantage? Comparing the modeling methods of token-level and trajectory-level rewards, it can help clarify the contribution of process-level supervision.

Q2. Comparison with Existing Contextual Faithfulness Methods.

Can the author compare CANOE[1] and Context-DPO[2] under the matching training and evaluation conditions? Since these methods also improve the context fidelity through training, compared to the CAD benchmark used solely in the inference stage, such comparisons will be more valuable for reference.

Q3. Contextual reliance and computational overhead.

Could the author provide controlled experiments, that is, modifying the relevant context while keeping the questions unchanged, to evaluate whether the model's response follows the modified evidence? Additionally, how much actual training time overhead will the context masking forward propagation introduce?

References

[1] Si et al., "Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning." AAAI 2026. https://doi.org/10.1609/aaai.v40i39.40582

[2] Bi et al., "Context-DPO: Aligning Language Models for Context-Faithfulness." ACL 2025. https://aclanthology.org/2025.findings-acl.536/

Reviewer_pmrTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the underuse of provided context in context-grounded tasks and proposes CORE. With the generated prefix fixed, the method compares the log-probabilities of generated tokens with the context available and masked, and uses their mean difference as an auxiliary reward gated by answer correctness for PPO/GRPO training. Experiments show improvements in accuracy and consistency with the provided context across multiple models and tasks.

Critical strengths
  1. The paper reports relatively consistent improvements across multiple models and training algorithms. The results on MemoTrap and other tasks involving knowledge conflicts are directly relevant to the problem studied.
  2. The method is simple and does not require additional process annotations or extra decoding operations at inference time. The accuracy gate is also supported by targeted ablations.
Critical weaknesses
  1. Token-level scoring has not been shown to provide fine-grained credit assignment. Although CORE computes contextual influence scores at the token level, these scores are still averaged into a trajectory-level reward before optimization. The paper does not show how the scores at individual positions are used to construct returns for the corresponding generation steps, nor does it explain the fine-grained credit assignment advantage over general trajectory-level reward shaping. Therefore, the meaning of dense process supervision and its relationship to the actual optimization procedure remain unclear.
  2. The relationship between sensitivity to context and contextual faithfulness has not been sufficiently validated. The log-probability difference with and without context reflects the sensitivity of generation probabilities to context availability, but does not itself determine whether the model correctly uses evidence relevant to the question. The accuracy gate only verifies the final answer and cannot verify intermediate reasoning. The MemoTrap and judge results provide valuable support, but do not sufficiently separate the contributions of increased contextual reliance, improved answer accuracy, and changes in output style.
Questions for authors
  1. The paper mentions Context-DPO in the related work but does not compare against it experimentally.
  2. The fixed generated prefix was produced with access to the context and may already contain key information from it. Even after the original context is masked, subsequent tokens may still obtain this information from the prefix. Thus, the log-probability difference more closely reflects the additional influence of the context given that prefix. How do the authors interpret the relationship between this log-probability difference and the model's actual contextual reliance in this situation?
  3. If the accuracy gate is retained, would using only the average log-probability conditioned on the context as a continuous reward yield similar gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.