- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies safety reward modeling when the appropriate preference depends on the evolving situation established by interaction history. It introduces ContextFlip, a benchmark of 1,432 matched families that hold the request and assistance/refusal responses fixed while changing the history to reverse the correct preference. Its STRICT-4 metric requires correctness under both situations and both candidate orders. The proposed ContextRM combines four-view preference supervision with recursive situational memory, followed by capability distillation and step-wise GRPO. Experiments report substantial gains on controlled long-history evaluations and additional transfer results on DynaBench.
Critical strengths
-
Important problem. Safety judgments should follow changes in permissions and constraints during long interactions. The benchmark provides a useful way to test whether reward models track these changes.
-
Clear presentation and strong results on ContextFlip. The paper is well organized and easy to read. The method achieves 95.91% STRICT-4, compared with 69.18% for the strongest reported baseline.
Critical weaknesses
-
Limited validation in realistic scenarios. The benchmark relies heavily on synthetic examples, and its 13 test conditions reuse the same 287 sample families. Performance in real agent workflows remains unclear. External results are less convincing: on the original DynaBench trajectory labels, the method's F1 is 75.07%, close to the published CSRM result of 75.80%.
-
Limited novelty and insufficient comparison with simpler methods. The diagnosed issues are common to reward models, and the solution mainly combines existing techniques. Stronger comparisons with simple summary-based methods are needed to establish its added value. The two-module system already achieves 94.16%, while the full RL model reaches 95.91%; this extra gain does not clearly justify the additional complexity.
Questions for authors
-
Clarification of the objective: What distinguishes Equation (5) from ordinary BT training on the same four counterfactual views with identical weighting and exposure? Does the reported BT ablation retain all four views? Please clarify whether the observed benefit comes from the objective, data coverage, or batching/sampling.
-
Evidence needed to assess pipeline necessity: Beyond the operation-level updater ablations in Table 2, can the authors provide matched end-to-end comparisons with a generic recursive summarizer and a clean-SFT updater without failure-guided augmentation, using the same scorer and comparable data and inference budgets? A paired comparison and cost analysis of the two-module system versus the final RL model would also help establish whether the additional stages are justified.
-
Clarification and evidence needed to assess transfer: How are the DynaBench response-level annotations validated independently of the teacher models used to construct and judge the counterfactual pairs? Is there existing evaluation on naturally occurring agent trajectories that supports the claimed practical benefit, including the false-positive/false-negative trade-off? Such evidence could materially change my assessment.
5 comments
Response(2/4)
Q1. Clarifying Where the Scorer's Gains Come From
We thank the reviewer for raising this question. Our method is consistent with the Bradley-Terry (BT) model in its objective formulation; the distinction lies in the training data structure and the corresponding loss aggregation scheme. The 4-view supervision can be understood as follows: the objective described in prior contextual preference alignment work [4] considers per-scenario supervision (essentially a 2-view setup that ignores candidate ordering effects). After observing the impact of long contexts on order stability, we further refined this by incorporating additional order sampling for each scenario, resulting in 4 views in total. Expanding from 2 views to 4 views can be viewed as data-augmented training designed to address the order shortcut problem discussed in Section 5.
When all 4 views are utilized, if each view is treated as an independent sample, the permissive/restrictive contexts and the two candidate orders of the same family are not guaranteed to fall within the same optimization update batch. Each gradient step is then formed solely by the currently sampled views; once certain views update the parameters, the remaining views in subsequent steps compute gradients against already-updated parameters. Even if the input order is arranged and the effective single-step batch size is matched to ours, mini-batch parameter updates during optimization would still alter the per-step gradient composition and optimization trajectory. In our implementation, the family batch size is 32, corresponding to a mini-batch size of 128 views (matching the exact number of views per update step). Therefore, our training can be regarded as BT training with data augmentation (4 views) and controlled input grouping (ensuring all views of the same family reside in the same optimization batch) at a batch size of 128.
To ensure a fair comparison on identical training data, the ablation baseline (Scorer-BT) also uses the same 4-view data to examine the order shortcut issue, differing only in how the loss is aggregated. Specifically, we first average the predicted scores across the two candidate orders within the same scenario and then compute and accumulate the logistic loss, thereby simulating the expected training objective of the 2-view setup. Consequently, the comparative gains of our scorer stem directly from explicit supervision over candidate order sampling.
Response(3/4)
W2/Q2. Clarify Design Motivation of the Updater and the Benefits of Decoupled Training and Single-Model Distillation
We would like to further clarify the novelty of our problem setting, which requires two key capabilities: compressing long interaction histories and making reliable, context-dependent safety judgments. To highlight the novelty and effectiveness of our approach, we discuss three aspects: comparisons with summarization baselines, the rationale for separate training followed by single-model distillation, and paired effectiveness and deployment-cost comparisons.
Failure-guided augmentation and Summarization baselines: The updater learns to retain currently valid information for safety judgments from histories where permission updates span multiple segments and repeatedly overwrite earlier states. Errors observed during updating guide our automated data augmentation.
To assess the benefit over general summarization, we supplement two summarization baselines. Prompt-only rolling summary uses base Qwen3-8B to recursively merge the previous summary with new inputs through prompting, following recursive summarization work [5]. SummLlama3.1-8B [6] is a dedicated 8B summarization model trained via DPO on summarization preference feedback. Both use the same recursive input construction and output budget as our updater and are paired with the same Scorer-Clean backend as ContextRM-System. Each cell below reports Accuracy / Strict-4 (%).
Table 3
| Method | 13-cell | Group B | Group C |
|---|---|---|---|
| Updater-0(clean-SFT) | 92.06 / 81.51 | 92.65 / 82.69 | 94.77 / 88.15 |
| Updater-Au | 93.12 / 85.18 | 94.05 / 86.06 | 49.04 / 6.27 |
| Updater-E3 | 97.80 / 94.16 | 97.79 / 94.31 | 97.74 / 94.08 |
| Prompt-only rolling summary | 70.68 / 40.39 | 69.48 / 37.63 | 73.17 / 46.69 |
| SummLlama3.1-8B | 58.44 / 18.63 | 62.92 / 28.92 | 56.53 / 17.07 |
Updater-0 (clean-SFT) is supervised fine-tuned solely on compact raw-context state data. Updater-Au is a standard length-augmented training control initialized from Updater-0, repeating existing training data for three epochs to match the additional training sample count and training pipeline of failure-guided augmentation. Updater-E3 is the final updater obtained through failure-guided data augmentation and used in ContextRM-System.
Comparisons with summarization methods demonstrate the clear benefit of our method in the setting of information preservation over complex, long histories. The difference between Updater-Au's performance on Group C and its performance on the other groups further validates the benefit of failure-guided training.
Motivation of Decomposition and Distillation: Prior research on context compression and refusal alignment [7] demonstrates that compressing context can weaken refusal alignment, making models more likely to assist requests that should be refused. Incorporating insights from prior studies on multi-task training interference (such as AdaMerging [8] and FuseChat [9]), we ultimately designed our overall framework by training two separate models: one for long-context information compression (updater) and another for making reliable safety judgments conditioned on the context (scorer), which are then combined to form ContextRM-System. ContextRM-System achieves 94.16% Strict-4, validating the effectiveness of isolating these two tasks during training. Subsequently, the primary role of OPD is to integrate both capabilities into a single model, mitigating negative transfer caused by multi-task interference while enabling a unified deployment form where one model executes both tasks. Experimental results also show a slight capability drop after merging; the subsequent RL stage then jointly optimizes the merged state updating and preference judgment via CoT-like step-level rewards.
Paired effectiveness and cost analysis of System/OPD/RL: The following paired comparisons assess capability integration and subsequent RL refinement.
Table 4
| Comparison | Strict-4 Difference (pp) | 95% Paired CI (pp) |
|---|---|---|
| OPD - System | -0.78 | [-3.99, +2.41] |
| RL - OPD | +2.53 | [-0.13, +4.66] |
System loads two 8B models (updater and scorer); OPD/RL loads one for both tasks. The RL stage can be viewed as the second training phase of OPD; although the 95% confidence interval slightly crosses zero, it yields overall positive gains. Regarding computational cost, the table below reports the post-load GPU memory footprint and the median loading latency across three runs (after warming up the filesystem cache) on a single RTX PRO 6000 (96GB) using an HF BF16 base model with LoRA:
Table 5
| Model | Loaded Models | Post-Load VRAM (GiB) | Loading Latency (s) |
|---|---|---|---|
| ContextRM-System | 2 | 32.15 | 8.07 |
| ContextRM-RL | 1 | 16.55 | 4.05 |
Response(1/4)
We thank the reviewer for recognizing the significance of our research problem. Below, we detail the external evaluation, clarify the roles of model merging and subsequent optimization, address the questions regarding the training units, and present evaluations on real agent trajectories.
We will clarify the external evaluation in the revised appendix as follows. CSRM illustrates the need for long-context processing: some policies exceed its 8,192-token window, so it splits them into items, retrieves the top 20 using BM25, and feeds them in their original order (Sec. 4.2 and Appendix C.1 of [1]). Our work addresses this challenge: reliable safety judgment requires both rule comprehension and faithful preservation of valid rules across long texts.
Furthermore, the DynaBench dataset exhibits a labeling inconsistency: some originally labeled PASS trajectories contain early turns that violate the rules before subsequent turns fulfill them. For example, in the original sample (source_index=458, 0-indexed), the overall trajectory is labeled PASS. Its second rule mandates that when discussing bioaugmentation to promote the degradation of chlorinated solvents (such as TCE) in anaerobic groundwater, the response must mention both suitable electron donors and favorable geochemical conditions. This dialogue consists of only two turns:
Table 1
Later compliance does not erase an earlier violation. Our method and CSRM judge each assistant turn using history up to that turn, then label a trajectory unsafe if any turn fails. This can disagree with original trajectory labels. With no public turn-level reference labels, we re-annotated the turns and found 136 original PASS trajectories with at least one FAIL turn (implementation in the anonymous repository; process in Q3). Since CSRM has not released weights or code, we compare DynaGuard and our model at the turn level, reporting FPR and FNR below:
Table 2
On the 407-trajectory subset excluding inconsistent trajectories, ContextRM-RL achieves 94.05% FAIL F1 versus DynaGuard's 76.40%, highlighting the effect of the aggregation discrepancy. On all 2,092 re-annotated turns, it achieves 84.09% versus 77.84%; FNR falls from 28.76% to 8.38%, while FPR rises from 3.96% to 8.81%. Our performance gains reflect fewer missed violations, with a slight increase in false positives. CSRM's F1 on all 543 original trajectories (75.80%) is close to ours (75.07%), but further metrics are unavailable. DynaBench inputs are relatively short (median 778 tokens; range 64-5,299); CSRM's training setup and BM25 policy retrieval help contextualize these results, paralleling the short-context behavior of our BM25 baseline in Table 3. Our primary contribution, however, lies in reliable contextual preference modeling over long contexts and complex interaction histories.
Following your suggestion, we added a real local agent tool-use pairwise evaluation based on AgentDojo [2] and tau-bench [3]: 25 tasks, 100 families, and six domains. Each family pairs permissive and restrictive histories with the request and both candidates fixed, so the correct preference flips with permission state. We test the original permissions and 4/8/16 additional updates; Q3 provides construction details and results.