acceptodds
Under review as a conference paper at ICLR 2027

RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback

Abstract

Reward models (RMs) are a key component of large language model post-training, providing reward signals for subsequent reinforcement learning. However, conventional discriminative RMs typically output only scalar scores, making it difficult to identify the response behaviors associated with their scoring decisions. Existing interpretation methods often rely on predefined high-level attributes and require repeated counterfactual interventions for each response pair to validate candidate explanations, lacking a closed-loop mechanism that uses RMs’ feedback to train a reusable explainer. To address this, we propose RewardExplainer, a framework that obtains feedback from the target reward model through counterfactual rewriting and uses this feedback to further optimize the explainer. RewardExplainer generates open-ended, atomic, and intervenable natural-language scoring mechanisms, making explanations more concrete, readable, and actionable. It further converts counterfactual feedback into preference supervision, enabling the explainer to more faithfully capture the target RM's scoring preferences and sensitive behaviors than single-pass generation. Extensive experiments across multiple target RMs and explainer backbones show consistent improvements. Beyond interpretation, we use the generated mechanisms to identify potential bias patterns and construct targeted debiasing data for fine-tuning the reward model, improving robustness on reward-hacking benchmarks. The code is available at: https://anonymous.4open.science/r/RewardExplainer/.

Primary area:
interpretability and explainable AI
Keywords:
Reward Models, Reward Model Interpretability, Preference Learning, Reward Hacking, Model Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EyBQOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RewardExplainer trains a reusable explainer to describe a fixed reward model’s preferences through atomic, editable natural-language mechanisms. Following supervised fine-tuning, bidirectional ADD/REMOVE interventions generate preference feedback for DPO. The resulting explanations also support bias discovery and targeted reward-model debiasing.

Critical strengths
  1. The clearest contribution is converting counterfactual verification into supervision for a reusable explainer, allowing direct explanation generation on unseen response pairs.
  2. DPO consistently improves the reported metrics across three reward models and three explainer backbones.
  3. The authors meaningfully address rewrite quality through evidence localization, applicability checks, and two evaluators.
Critical weaknesses
  1. Counterfactual reward explanations and automatic bias discovery already exist. The main comparisons lack feedback-budget-matched candidate search and SFT on verified-success mechanisms. Consequently, it remains unclear whether DPO learns substantially better explanations than simpler ways of exploiting the same generated candidates and reward queries.

  2. The paper motivates amortizing repeated explanation search, but does not establish the target-model query savings, inference cost, or break-even point relative to per-example search at matched explanation quality. Reported training times alone do not resolve this question.

  3. The explainer receives responses already identified as higher and lower reward. The metric therefore primarily measures whether its explanations consistently characterize the supplied preference direction, rather than independently predicting the reward model’s preference.

  4. The novelty discussion should address closer prior work. SAEExplainer (2026) already follows an SFT–candidate generation–target-model feedback–DPO pipeline for training a reusable explainer, albeit for SAE features rather than reward preferences. Multi-Domain Explainability of Preferences (2025) and Obj-Disco (2026) also establish automated, open-ended explanations of preference mechanisms and alignment objectives. These works do not subsume the full method, but narrow its contribution to bidirectional counterfactual supervision for reward-model explanations. Please clarify what additional methodological challenges and empirical benefits this formulation provides.

Questions for authors

See the weakness

10 comments
AuthorsOpenReview ↗

Response to Reviewer EyBQ (Part 1)

We thank the reviewer for the thoughtful comments and discussion.

1. Counterfactual reward explanations and automatic bias discovery already exist. The main comparisons lack feedback-budget-matched candidate search and SFT on verified-success mechanisms. Consequently, it remains unclear whether DPO learns substantially better explanations than simpler ways of exploiting the same generated candidates and reward queries.

We agree that comparisons with conventional SFT alone are insufficient to fully establish the advantages of DPO. Therefore, we conduct three additional experiments to further distinguish DPO optimization from simple candidate search and supervised training on verified-success mechanisms.

(1) Candidate Generation Efficiency

First, we compare the BOTH success rates of the SFT model, which generates 12 candidate mechanisms followed by deduplication and verification, with the DPO-trained model, which generates only 3 candidate mechanisms per sample. We use FsfairX as target reward model and Qwen3-4B as RewardExplainer backbone.

StageAvg. Candidate Mechanisms per SampleCandidate-level BOTH Success
SFT Candidate Generation8.6225.69%
DPO-trained Model3.0048.44%

The results show that the DPO-trained model achieves a substantially higher BOTH success rate among its three generated mechanisms than the SFT model, which retains an average of 8.62 candidates per sample after deduplication. This provides preliminary evidence that RewardExplainer can generate a higher proportion of effective explanatory mechanisms with fewer candidates, rather than relying on generating and subsequently filtering a large candidate pool.

(2) Budget-matched Candidate Search

We further conduct a budget-matched candidate search experiment on the same 300 held-out test samples, with both methods generating 3 candidate mechanisms per sample.

MethodCandidate-level BOTH SuccessBest-of-3 BOTH Success
SFT Candidate Search32.33%62.33%
RewardExplainer (DPO)48.44%77.33%

Here, Best-of-3 Success denotes the proportion of test samples for which at least one of the three candidate mechanisms achieves BOTH success. Even under the same candidate generation and reward feedback budgets, DPO achieves higher BOTH success (+16.11 pp) and Best-of-3 success (+15.00 pp). This indicates that the advantage of DPO does not simply come from generating more candidates or making more RM queries.

(3) Verified-success SFT

Finally, to further investigate whether DPO outperforms directly using successful mechanisms for supervised fine-tuning, we construct a Verified-success SFT baseline by training on mechanisms that have passed BOTH counterfactual validation, and evaluate it on the same independent test set.

MetricVerified-success SFTDPO
Mechanism Quality1.3701.637
Preference Prediction84.64%91.33%
ADD Success55.06%64.33%
REMOVE Success53.56%60.00%
BOTH Success41.20%48.67%
Best-of-371.00%77.00%

Although directly training on verified-success mechanisms with SFT already yields substantial improvements, DPO achieves better results across all evaluation metrics. This suggests that the advantage of DPO lies not only in learning from verified-success mechanisms, but also in further improving the quality of explanations and their alignment with the target reward model's preferences through preference optimization.

2. The paper motivates amortizing repeated explanation search, but does not establish the target-model query savings, inference cost, or break-even point relative to per-example search at matched explanation quality. Reported training times alone do not resolve this question.

Thank you for your question. We address these concerns as follows.

(1) Target-model Query Savings

The existing reward model explanation work Interpreting Language Reward Models via Contrastive Explanations serves as a baseline for per-example counterfactual search. This method defines 15 evaluation attributes and generates perturbations along each attribute for both the chosen and rejected responses, resulting in 30 perturbed candidates per response pair. These candidates are then scored by the target RM to identify counterfactual explanations that reverse the original preference.

In contrast, RewardExplainer concentrates RM feedback in the offline training stage. Taking Skywork as an example, we use 4,000 response pairs to construct DPO training data, generating 26,908 candidate mechanisms (6.73 per pair on average). Each mechanism undergoes both ADD and REMOVE counterfactual interventions and validation.

[1] Jiang, J., Bewley, T., Mishra, S., Lécué, F., & Veloso, M. (2025). Interpreting Language Reward Models via Contrastive Explanations. ICLR 2025

AuthorsOpenReview ↗

Response to Reviewer EyBQ (Part 2)

(2) Inference Cost

After DPO training, RewardExplainer requires only a single batched generation call to produce explanatory mechanisms for each response pair. Our goal is to enable the model to directly generate high-quality explanations faithful to the target RM's preferences, without requiring additional per-example candidate selection or RM verification. Thus, during inference, RewardExplainer requires only one generation call per pair, whereas per-example search methods must generate multiple counterfactual candidates and evaluate them using the target RM. This eliminates repeated online RM queries by shifting the feedback cost to the offline training stage.

To further quantify online inference efficiency, we measure the average online latency per response pair. RewardExplainer requires 2.84 seconds/pair, compared with 7.08 seconds/pair for per-example search. This represents a 59.89% reduction in average online latency, corresponding to approximately 2.49× inference speedup.

(3) Break-even Point

Assuming that per-example search requires 30 target RM scoring instances per response pair, the RM-query break-even point of RewardExplainer is:

NBE=5381630≈1794N_{\mathrm{BE}} = \frac{53816}{30} \approx 1794

This means that after approximately 1,794 response pairs, the cumulative target RM scoring cost of RewardExplainer becomes lower than that of the per-example search baseline. For 4,000 subsequent explanation requests, RewardExplainer is estimated to save 66,184 target RM scoring instances, corresponding to a 55.15% reduction in cumulative RM scoring cost.

3. The explainer receives responses already identified as higher and lower reward. The metric therefore primarily measures whether its explanations consistently characterize the supplied preference direction, rather than independently predicting the reward model’s preference.

We would like to further clarify the task formulation of RewardExplainer. Our work focuses on Post-hoc Reward Model Explanation, rather than Independent Reward Preference Prediction. Specifically, given a response pair whose preference ordering has already been established by the target reward model, our goal is to identify the underlying behavioral mechanisms that explain the reward model's preferences and scoring differences. Therefore, RewardExplainer aims to provide explanations based on the target reward model's established preferences, rather than independently predicting preferences without access to the reward model's decisions.

We use an LLM judge to determine which response better matches the generated mechanism and evaluate whether its judgment is consistent with the target reward model's preference. We acknowledge that this metric measures the consistency between the mechanisms and the known preference direction, rather than the explainer's independent preference prediction capability. Moreover, this metric complements our bidirectional counterfactual verification, which examines whether adding the behavior described by a mechanism to the lower-reward response, or removing it from the higher-reward response, leads to the expected changes in the target reward model's scores. Thus, the former evaluates preference consistency, while the latter assesses counterfactual validity. We will further clarify this distinction and refine the description of the relevant metric in the revised manuscript to avoid potential ambiguity.

AuthorsOpenReview ↗

Response to Reviewer EyBQ (Part 3)

4. The novelty discussion should address closer prior work. SAEExplainer (2026) already follows an SFT–candidate generation–target-model feedback–DPO pipeline for training a reusable explainer, albeit for SAE features rather than reward preferences. Multi-Domain Explainability of Preferences (2025) and Obj-Disco (2026) also establish automated, open-ended explanations of preference mechanisms and alignment objectives. These works do not subsume the full method, but narrow its contribution to bidirectional counterfactual supervision for reward-model explanations. Please clarify what additional methodological challenges and empirical benefits this formulation provides.

We acknowledge their connections to RewardExplainer and clarify our contributions as follows. SAEExplainer adopts a similar feedback-driven DPO training framework but focuses on SAE feature explanations using activation feedback. In contrast, RewardExplainer targets pairwise reward preferences and introduces bidirectional counterfactual interventions to validate behavioral mechanisms through actual reward score changes. Multi-Domain Explainability of Preferences focuses on concept-based preference analysis, while Obj-Disco discovers implicit alignment objectives from model behaviors and training trajectories. Our work instead focuses on local, actionable explanations of reward model preferences. Furthermore, RewardExplainer goes beyond explanation by leveraging the discovered mechanisms to identify potential reward model biases, construct targeted preference training data, and fine-tune the target reward model. Thus, our contributions lie in bidirectional counterfactual supervision for reward model explanations and explanation-guided reward model improvement. We will clarify these distinctions and expand the related work discussion in the revised manuscript.

Reviewer_G3JeOpenReview ↗
Rating
4
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to explain why a trained reward model prefers one response over another. The proposed method generates editable, natural-language explanations for pairs of responses—one with a high reward score and one with a low score—and then subjects these explanations to a bidirectional counterfactual verification: incorporating the described behavior into the low-scoring response to see if its reward score increases, and removing that behavior from the high-scoring response to see if its reward score decreases. Explanations that pass this bidirectional counterfactual verification are used to construct preference data, which then trains a reusable "explainer" model via Direct Preference Optimization (DPO).

Furthermore, the authors identify behaviors within these explanations that may reflect biases in the reward model, use these behaviors to rewrite the preference training data, and perform targeted fine-tuning of the reward model. Experiments conducted across three reward models and three backbone models for the explainer support the conclusion that training an explainer using feedback from the target reward model improves the success rate of its explanations—as measured by the authors' Counterfactual Faithfulness evaluation—and yields performance gains on certain downstream bias benchmarks.

Excellent work. I recommend acceptance.

Critical strengths
  1. The connection between the method and the task is well-defined; the explanations are subjected to empirical validation based on the actual behavior of the target reward model. This work goes beyond merely prompting a language model to generate plausible-sounding explanations; it validates candidate mechanisms by measuring the actual shifts in the target reward model’s scores, and subsequently uses this feedback for training. Leveraging known pairwise reward-model preference labels is a sound setup for the task of post-hoc explanation.

  2. The core improvements remain consistent across various combinations of reward models and explainers. In the nine combinations presented in Table 1, the DPO-based versions consistently outperform the SFT-only baselines across the listed metrics.

Critical weaknesses
  1. The assessment of "Counterfactual Faithfulness" relies primarily on the direction of the score change, which does not sufficiently account for incidental changes introduced during the rewriting process. "ADD" success requires only an increase in reward, while "REMOVE" success requires only a decrease; both marginal and substantial shifts are counted as successes. Consequently, the current metrics do not clarify to what extent the proposed mechanism actually accounts for the original reward difference. Furthermore, natural language rewriting may inadvertently alter factors such as text length, phrasing, or other attributes. The Random Perturbation baseline included in the appendix does not isolate the reward change attributable to the target mechanism itself, leaving the potential influence of other rewriting effects unaddressed.

*Updated on 18th November 2026: Solved.

Questions for authors
  1. What is the magnitude of the reward changes observed in the ADD/REMOVE interventions? Could a control experiment—in which the target mechanism is held constant during rewriting—or manual verification be used to confirm that the observed success is indeed driven by the intended mechanism?
3 comments
AuthorsOpenReview ↗

Response to Reviewer G3Je

We thank the reviewer for your positive feedback and insightful suggestions.

1. What is the magnitude of the reward changes observed in the ADD/REMOVE interventions?

To address this concern, we use FsfairX as the target RM and Qwen3-4B as the base explainer. We compare the reward score changes induced by counterfactual edits using GPT-4o explanations, RewardExplainer (SFT), and RewardExplainer (DPO). The experimental results are presented below:

MethodADD (Mean Increase)REMOVE (Mean Decrease)Mean Directional Change
RewardExplainer (DPO)+0.776+0.619+0.698
RewardExplainer (SFT)+0.467+0.405+0.436
Contrastive Explanations (GPT-4o)+0.487+0.466+0.477

As shown in the table, RewardExplainer (DPO) achieves larger mean reward changes in both the ADD and REMOVE directions. Its mean directional change reaches 0.698, outperforming both the SFT and GPT-4o baselines. These results indicate that the mechanisms optimized through DPO not only induce reward changes in the expected directions but also lead to larger changes in reward magnitude.

Furthermore, to evaluate our method under stricter success criteria, we introduce different reward change thresholds (τ = 0, 0.1, 0.2, 0.5, 1.0) and recompute the ADD, REMOVE, and BOTH success rates. Specifically, an ADD intervention is considered successful only if the induced reward increase exceeds the specified threshold, while a REMOVE intervention is considered successful only if the induced reward decrease exceeds the same threshold. A BOTH intervention is considered successful only when both conditions are satisfied. All methods are evaluated on 300 unseen samples.

The experimental results are presented below (each entry represents ADD / REMOVE / BOTH success rates, in %):

ThresholdRewardExplainer (DPO)RewardExplainer (SFT)GPT-4o
> 065.33 / 58.56 / 48.4445.00 / 42.56 / 32.3345.33 / 43.22 / 35.33
> 0.161.22 / 55.67 / 43.1141.33 / 41.00 / 29.6742.44 / 40.56 / 32.00
> 0.255.00 / 52.00 / 37.1137.89 / 38.22 / 25.4438.56 / 36.89 / 26.89
> 0.540.89 / 40.22 / 22.6728.22 / 28.22 / 16.5626.56 / 26.00 / 15.22
> 1.025.89 / 27.11 / 11.4417.33 / 17.56 / 7.8918.44 / 18.22 / 8.78

The results show that RewardExplainer (DPO) consistently achieves the highest ADD, REMOVE, and BOTH success rates across all tested thresholds, even under increasingly stringent reward change criteria. Although the absolute success rates decrease as the threshold increases for all methods, the advantage of DPO persists across different thresholds.

2.Could a control experiment—in which the target mechanism is held constant during rewriting—or manual verification be used to confirm that the observed success is indeed driven by the intended mechanism?

To address the lack of human validation and the concern that some edits may alter content beyond the target mechanism, we conducted an additional human evaluation of counterfactual rewriting quality along three dimensions: Mechanism Realization, Edit Precision, and Content Preservation. We randomly sampled 50 examples, and two annotators independently rated the ADD and REMOVE rewrites on a scale of 1–10. The results are shown below:

AnnotatorADDREMOVE
Annotator A9.509.10
Annotator B9.109.10
Average9.309.10

The overall average score is 9.2. These results indicate that the generated counterfactual rewrites generally realize the intended mechanisms while maintaining high editing precision and preserving content unrelated to the target mechanisms as much as possible.

Thank you for your reply. The threshold analysis addressed my concerns. I keep my score.

Thank you for your valuable suggestions. We sincerely appreciate your constructive feedback and will incorporate these changes into the final version of our paper.

Reviewer_7BdwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RewardExplainer, a small language model trained to explain why a reward model scores one response higher than another. Given a prompt and two responses, the explainer outputs a few short natural-language descriptions of response behaviors (called "mechanisms"), such as "uses bullet points." Training has two stages. First, the explainer is fine-tuned on mechanisms written by GPT-4o. Second, it samples candidate mechanisms, and GPT-5.4-mini rewrites the responses to add each behavior to the lower-scored response and remove it from the higher-scored one. Mechanisms for which both edits move the reward in the expected direction are used as preferred outputs for DPO. The trained explainer is then used to find possible reward-model biases, GPT-4o rewrites rejected responses to contain these biases, and the reward model is fine-tuned on the resulting pairs. For explanations, RewardExplainer is compared with two baselines (direct generation by GPT-4o, and the contrastive explanations of Jiang et al. 2024) on three reward models with three explainer backbones. It scores higher than both on all three metrics (the success rate of the add and remove edits, explanation quality rated by an LLM judge, and an LLM's accuracy in using a mechanism to pick the higher-scored response) for every reward model and backbone. For debiasing, the fine-tuned reward models are compared with the original models and with a control fine-tuned on the same amount of general preference data; they score higher on two reward-hacking benchmarks (RewardHackBench and JudgeBias) for all three reward models, while general accuracy on JudgeBench stays roughly the same.

Critical strengths
  • Explaining reward models in open-ended natural language, and using the explanations to improve the reward model, is a meaningful direction. This paper closes the loop from interpretation to model improvement and I think such a contribution is timely for the literature.
Critical weaknesses
  • The quality of the explanations is not validated with human evaluations, and the automatic metrics do not show that the explanations reflect the reward model. All three metrics rely on LLMs, and no human evaluation / validation is reported. The edit-success metric is the same test used to create the DPO training signal, and it seems weak: a success only needs the reward to move in the expected direction by any amount, whereas Jiang et al. (2024) require the preference to flip, and there is no control showing what random or mismatched mechanisms would score. The LLM judge does not see the reward scores, so it checks whether the mechanisms describe real differences between the two responses, not whether they explain the reward model. The preference-prediction metric asks an LLM which response better matches a mechanism, but mechanisms are written to describe what the higher-scored response does more, so this largely checks direction rather than the reward model. The edits themselves are checked only by LLM judges, and some appear to change more than the stated behavior (Example 2 in Appendix C).
  • The debiasing results do not show that the explainer is what drives the improvement. The six biases reported for RM-Mistral (verbosity, formal style, restating context, broadening scope, confident wording, added uncertainty statements) are already well known, and there is no ablation that builds the same debiasing data from a list of known biases, from the attributes of previous works or from biases proposed directly by GPT-4o. The debiasing results come from a single run without variance estimates, the gains over the control are small for one reward model (+0.4 on RewardHackBench and +0.3 on JudgeBias for Skywork), and JudgeBench accuracy falls slightly below the original model for two of the three reward models.
Questions for authors
  • Can you report a human evaluation on a sample of explanations, judging whether each mechanism is correct and useful, and measure how well the LLM judge agrees with people?
  • What success rates do random mechanisms, or mechanisms taken from a different response pair, achieve under the same edit test?
  • How do the explanation metrics separate explaining the reward model from describing the differences between two responses? For example, how do mechanisms written without any access to the reward model score on the LLM judge and on preference prediction?
  • Does the debiased reward model also improve on bias types that were not injected, and why does JudgeBench accuracy fall below the original model for two of the three reward models?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 7Bdw (Part 1)

We thank the reviewer for the helpful comments.

1. The quality of the explanations is not validated with human evaluations, and the automatic metrics do not show that the explanations reflect the reward model. All three metrics rely on LLMs, and no human evaluation / validation is reported. The edit-success metric is the same test used to create the DPO training signal, and it seems weak: a success only needs the reward to move in the expected direction by any amount, whereas Jiang et al. (2024) require the preference to flip, and there is no control showing what random or mismatched mechanisms would score. The LLM judge does not see the reward scores, so it checks whether the mechanisms describe real differences between the two responses, not whether they explain the reward model. The preference-prediction metric asks an LLM which response better matches a mechanism, but mechanisms are written to describe what the higher-scored response does more, so this largely checks direction rather than the reward model. The edits themselves are checked only by LLM judges, and some appear to change more than the stated behavior (Example 2 in Appendix C). (Weakness 1)

We thank the reviewer for these detailed and constructive comments. In response, we have conducted additional human evaluations and counterfactual editing experiments under different thresholds. We have also clarified the design objectives and scope of our original evaluation metrics.

(1) Human Evaluation

To address the lack of human validation and the concern that some edits may alter content beyond the target mechanism, we conducted an additional human evaluation of counterfactual rewriting quality along three dimensions: Mechanism Realization, Edit Precision, and Content Preservation. We randomly sampled 50 examples, and two annotators independently rated the ADD and REMOVE rewrites on a scale of 1–10. The results are shown below:

AnnotatorADDREMOVE
Annotator A9.509.10
Annotator B9.109.10
Average9.309.10

The overall average score is 9.2. These results indicate that the generated counterfactual rewrites generally realize the intended mechanisms while maintaining high editing precision and preserving content unrelated to the target mechanisms as much as possible.

(2) Stricter Edit-success Evaluation

We agree that the original Edit-success metric, which only requires the Reward Score to change in the expected direction, may not sufficiently reflect the magnitude of the intervention effect. To address this concern, we further evaluate ADD, REMOVE, and BOTH success rates using five different Reward Change Thresholds: 0, 0.1, 0.2, 0.5, and 1.0 on rm FsfairX-LLaMA3-RM-v0.1. This allows us to examine the performance of different methods under increasingly stringent success criteria. The results are presented below:

ThresholdRewardExplainer (DPO)RewardExplainer (SFT)GPT-4oContrastive Explanations
> 065.33 / 58.56 / 48.4445.00 / 42.56 / 32.3345.33 / 43.22 / 35.3327.22 / 28.11 / 20.89
> 0.161.22 / 55.67 / 43.1141.33 / 41.00 / 29.6742.44 / 40.56 / 32.0024.44 / 27.78 / 18.67
> 0.255.00 / 52.00 / 37.1137.89 / 38.22 / 25.4438.56 / 36.89 / 26.8921.22 / 27.22 / 16.00
> 0.540.89 / 40.22 / 22.6728.22 / 28.22 / 16.5626.56 / 26.00 / 15.2215.44 / 25.22 / 11.56
> 1.025.89 / 27.11 / 11.4417.33 / 17.56 / 7.8918.44 / 18.22 / 8.7810.11 / 20.89 / 6.11

As the Reward Change Threshold increases, the success rates of all methods naturally decrease. However, RewardExplainer (DPO) consistently achieves the best results across all thresholds. These results demonstrate that the advantage of DPO is not merely attributable to marginal changes in Reward Scores and remains evident under more stringent success criteria.

AuthorsOpenReview ↗

Response to Reviewer 7Bdw (Part 2)

(3) Clarification of LLM-based Evaluation

Regarding the concern that the LLM Judge does not have access to Reward Scores, we would like to clarify that we intentionally withhold the target RM scores only for the Preference Prediction metric. This metric aims to examine whether the generated mechanisms are grounded in observable textual evidence and can accurately capture behavioral differences between two responses without access to reward scores. Therefore, it primarily measures the consistency between mechanism descriptions and response content, rather than directly establishing the faithfulness of explanations to the Reward Model's preferences. Furthermore, to validate the reliability of Preference Prediction, we conducted an additional human evaluation on 50 examples for RM-Mistral-7B, comparing the choices made by two human annotators with those of the LLM Judge. The average agreement rate between the human annotators and the LLM Judge reached 93.00%, indicating strong consistency between LLM-based judgments and human assessments. This provides additional human validation for the Preference Prediction metric.

For the other two evaluation metrics, we provide the target RM scores as context because our goal is to assess whether the generated mechanisms can explain the Reward Model's preference behavior. Specifically, the LLM Judge evaluates whether a mechanism can reasonably explain the model's scoring preferences based on the response content, mechanism description, and RM scores. Meanwhile, the counterfactual edit-success metric directly measures changes in the target RM scores before and after editing to examine whether the mechanism intervention produces the expected reward changes. Therefore, these three metrics provide complementary evaluations of the generated mechanisms from different perspectives: textual consistency, preference explanation, and counterfactual intervention effectiveness.

2. The debiasing results do not show that the explainer is what drives the improvement. The six biases reported for RM-Mistral (verbosity, formal style, restating context, broadening scope, confident wording, added uncertainty statements) are already well known, and there is no ablation that builds the same debiasing data from a list of known biases, from the attributes of previous works or from biases proposed directly by GPT-4o. The debiasing results come from a single run without variance estimates, the gains over the control are small for one reward model (+0.4 on RewardHackBench and +0.3 on JudgeBias for Skywork), and JudgeBench accuracy falls slightly below the original model for two of the three reward models.

(1) Comparison with Previous-Work-Based Debiasing.

To investigate whether the debiasing improvements can be achieved simply by leveraging previously identified bias types, we introduce an additional baseline based on Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models. Specifically, we adopt its five bias types and corresponding debiasing data (Length, Structure, Jargon, Sycophancy, Vagueness). We mix the corresponding debiasing data with general preference data at a 1:1 ratio to fine-tune FsfairX-LLaMA3-RM-v0.1. We then evaluate the resulting model on RewardHackBench. The results are presented below:

MethodOverall Avg. ↑Surface ↑Reason ↑Sycophancy ↑Off-topic ↑Style ↑
Base70.2281.8277.1468.5753.5474.61
General74.7783.3377.1466.1969.4779.79
Baseline75.7089.9077.1469.0567.7076.68
RewardExplainer (Ours)77.8485.8681.4372.3873.0178.76

As shown in the table, the Baseline improves the overall accuracy from 74.77 to 75.70, demonstrating that previously identified bias types can indeed provide useful debiasing signals. However, RewardExplainer achieves a higher overall accuracy of 77.84. These results suggest that the improvements brought by RewardExplainer cannot be fully explained by simply incorporating previously known bias types. Instead, identifying and validating mechanisms specific to the target reward model provides additional benefits for reward model debiasing.

[1] Bharadwaj, A., Malaviya, C., Joshi, N., & Yatskar, M. (2026). Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer 7Bdw (Part 3)

(2) Multi-seed Experiments and JudgeBench analysis.

To evaluate the stability of our debiasing results, we retrained both the General baseline and RewardExplainer using three random seeds (42, 123, and 2026). The results show that our method consistently achieves better mean performance than the General baseline on both RewardHackBench and JudgeBiasBench across all three reward models. We report the mean and standard deviation over the three runs below.

Target RMMethodOverallSurfaceReason.Sycoph.Off-topicStyleJ-BiasJ-Bench
FsfairXBase70.2281.8277.1468.5753.5474.6162.5958.57
General74.56±0.3683.50±1.2776.90±0.4165.87±0.2769.03±0.4479.62±0.3065.13±0.0259.43±0.00
Ours77.87±0.1085.86±0.0081.67±0.4172.22±0.7372.71±0.2679.10±0.3070.45±0.1560.38±0.16
RM-Mistral-7BBase70.2281.3177.8677.6247.3572.0269.0463.51
General76.01±0.2785.35±1.0179.52±1.0972.06±0.5569.17±0.2676.17±0.5270.16±0.1861.90±0.59
Ours78.49±0.2182.32±0.8784.05±0.4179.05±0.4873.75±0.5175.47±0.6074.12±0.2662.00±0.00
SkyworkBase76.9479.2977.8679.5269.0380.3175.7770.69
General77.63±0.3279.97±0.2978.33±0.8276.19±0.4874.63±0.2679.79±0.5276.04±0.1871.24±0.33
Ours77.97±0.3180.30±0.8777.62±0.4177.30±0.2775.22±0.4479.79±0.0076.50±0.1070.76±0.44

(3) Performance Analysis and JudgeBench Evaluation.

Regarding Skywork, we attribute its relatively modest improvement partly to its stronger initial performance, which may leave less room for further debiasing gains. Nevertheless, our multi-seed results show consistent improvements over the General baseline on both RewardHackBench and J-Bias, suggesting that the gains are stable across different runs.

We additionally evaluate the models on JudgeBench, which assesses the general judging capabilities of reward models across four domains: knowledge, reasoning, mathematics, and coding. This evaluation allows us to examine whether debiasing compromises the models' general judgment capabilities. Our primary objective is to improve performance on the two bias-oriented benchmarks while maintaining the general judging capabilities of the original reward models. We acknowledge this remaining trade-off and will discuss it in the revised manuscript.

3. What success rates do random mechanisms, or mechanisms taken from a different response pair, achieve under the same edit test?

To address the question about random mechanisms or mechanisms taken from different response pairs, we conduct a mismatched-mechanism experiment on 100 held-out pairs. Specifically, we randomly assign DPO-generated mechanisms from other response pairs while keeping the ADD/REMOVE editing and evaluation procedures unchanged. The results are as follows:

MethodADD ↑REMOVE ↑BOTH ↑Best-of-3 ↑
Matched DPO64.33%60.00%48.67%77.00%
Mismatched15.33%17.00%9.33%21.00%

Our method substantially outperforms the mismatched baseline, particularly on BOTH success (48.67% vs. 9.33%) and Best-of-3 (77.00% vs. 21.00%). These results indicate that counterfactual success is not merely caused by arbitrary edits, but depends strongly on the alignment between mechanisms and response pairs, supporting RewardExplainer's ability to capture the target RM's preference behavior.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Responses To All Reviewers

We sincerely thank all reviewers for their careful evaluation and valuable, constructive feedback. We are pleased that the reviewers recognized the motivation, methodological design, and potential value of RewardExplainer. In particular, Reviewers 7Bdw, G3Je, and EyBQ appreciated our approach of leveraging feedback from the target reward model as training supervision and optimizing the explainer through DPO. Reviewer G3Je highlighted the clear connection between our method and the task and recommended acceptance. In addition, Reviewer EyBQ acknowledged our experiments across different reward models and explainer backbones, as well as our evaluation of counterfactual editing quality.

In the following responses, we clarify misunderstandings and explain how we plan to incorporate the reviewers' valuable suggestions into the revised manuscript. We sincerely appreciate their feedback, which has helped us significantly improve the quality of our work.

  • We add human evaluation to further validate counterfactual rewriting quality and assess the agreement between human annotators and the LLM Judge on the Preference Prediction task.

  • We conduct counterfactual validation under different reward-change thresholds, demonstrating that RewardExplainer consistently outperforms the baselines even under stricter success criteria.

  • We introduce mechanism-matching and reward-feedback-free control experiments, including a Mismatched-mechanism Control and mechanism generation without reward scores or preference labels, to further examine the relationship between generated mechanisms and the target reward model's preferences.

  • We provide additional comparisons between DPO, candidate search, and Verified-success SFT, further validating the contribution of DPO optimization.

  • We conduct multi-seed experiments to evaluate the stability and reliability of our results.

  • We provide additional inference-efficiency and query-cost analyses to quantify RewardExplainer's advantages over per-example counterfactual search in terms of online inference efficiency and the cost break-even point.

  • We introduce an additional baseline based on previously identified bias types to assess the added value and generalization of the mechanisms discovered by RewardExplainer beyond existing bias categories.

  • We further clarify our task formulation and evaluation metrics, and expand our discussion of how RewardExplainer differs from prior work and its methodological contributions.

We sincerely thank all reviewers again for their valuable feedback. We believe these additional experiments and clarifications strengthen the experimental rigor of RewardExplainer, clarify its scope and core contributions, and improve the overall completeness and clarity of the manuscript.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.