- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RewardExplainer trains a reusable explainer to describe a fixed reward model’s preferences through atomic, editable natural-language mechanisms. Following supervised fine-tuning, bidirectional ADD/REMOVE interventions generate preference feedback for DPO. The resulting explanations also support bias discovery and targeted reward-model debiasing.
Critical strengths
- The clearest contribution is converting counterfactual verification into supervision for a reusable explainer, allowing direct explanation generation on unseen response pairs.
- DPO consistently improves the reported metrics across three reward models and three explainer backbones.
- The authors meaningfully address rewrite quality through evidence localization, applicability checks, and two evaluators.
Critical weaknesses
-
Counterfactual reward explanations and automatic bias discovery already exist. The main comparisons lack feedback-budget-matched candidate search and SFT on verified-success mechanisms. Consequently, it remains unclear whether DPO learns substantially better explanations than simpler ways of exploiting the same generated candidates and reward queries.
-
The paper motivates amortizing repeated explanation search, but does not establish the target-model query savings, inference cost, or break-even point relative to per-example search at matched explanation quality. Reported training times alone do not resolve this question.
-
The explainer receives responses already identified as higher and lower reward. The metric therefore primarily measures whether its explanations consistently characterize the supplied preference direction, rather than independently predicting the reward model’s preference.
-
The novelty discussion should address closer prior work. SAEExplainer (2026) already follows an SFT–candidate generation–target-model feedback–DPO pipeline for training a reusable explainer, albeit for SAE features rather than reward preferences. Multi-Domain Explainability of Preferences (2025) and Obj-Disco (2026) also establish automated, open-ended explanations of preference mechanisms and alignment objectives. These works do not subsume the full method, but narrow its contribution to bidirectional counterfactual supervision for reward-model explanations. Please clarify what additional methodological challenges and empirical benefits this formulation provides.
Questions for authors
See the weakness
10 comments
Response to Reviewer EyBQ (Part 2)
(2) Inference Cost
After DPO training, RewardExplainer requires only a single batched generation call to produce explanatory mechanisms for each response pair. Our goal is to enable the model to directly generate high-quality explanations faithful to the target RM's preferences, without requiring additional per-example candidate selection or RM verification. Thus, during inference, RewardExplainer requires only one generation call per pair, whereas per-example search methods must generate multiple counterfactual candidates and evaluate them using the target RM. This eliminates repeated online RM queries by shifting the feedback cost to the offline training stage.
To further quantify online inference efficiency, we measure the average online latency per response pair. RewardExplainer requires 2.84 seconds/pair, compared with 7.08 seconds/pair for per-example search. This represents a 59.89% reduction in average online latency, corresponding to approximately 2.49× inference speedup.
(3) Break-even Point
Assuming that per-example search requires 30 target RM scoring instances per response pair, the RM-query break-even point of RewardExplainer is:
This means that after approximately 1,794 response pairs, the cumulative target RM scoring cost of RewardExplainer becomes lower than that of the per-example search baseline. For 4,000 subsequent explanation requests, RewardExplainer is estimated to save 66,184 target RM scoring instances, corresponding to a 55.15% reduction in cumulative RM scoring cost.
3. The explainer receives responses already identified as higher and lower reward. The metric therefore primarily measures whether its explanations consistently characterize the supplied preference direction, rather than independently predicting the reward model’s preference.
We would like to further clarify the task formulation of RewardExplainer. Our work focuses on Post-hoc Reward Model Explanation, rather than Independent Reward Preference Prediction. Specifically, given a response pair whose preference ordering has already been established by the target reward model, our goal is to identify the underlying behavioral mechanisms that explain the reward model's preferences and scoring differences. Therefore, RewardExplainer aims to provide explanations based on the target reward model's established preferences, rather than independently predicting preferences without access to the reward model's decisions.
We use an LLM judge to determine which response better matches the generated mechanism and evaluate whether its judgment is consistent with the target reward model's preference. We acknowledge that this metric measures the consistency between the mechanisms and the known preference direction, rather than the explainer's independent preference prediction capability. Moreover, this metric complements our bidirectional counterfactual verification, which examines whether adding the behavior described by a mechanism to the lower-reward response, or removing it from the higher-reward response, leads to the expected changes in the target reward model's scores. Thus, the former evaluates preference consistency, while the latter assesses counterfactual validity. We will further clarify this distinction and refine the description of the relevant metric in the revised manuscript to avoid potential ambiguity.
Response to Reviewer EyBQ (Part 3)
4. The novelty discussion should address closer prior work. SAEExplainer (2026) already follows an SFT–candidate generation–target-model feedback–DPO pipeline for training a reusable explainer, albeit for SAE features rather than reward preferences. Multi-Domain Explainability of Preferences (2025) and Obj-Disco (2026) also establish automated, open-ended explanations of preference mechanisms and alignment objectives. These works do not subsume the full method, but narrow its contribution to bidirectional counterfactual supervision for reward-model explanations. Please clarify what additional methodological challenges and empirical benefits this formulation provides.
We acknowledge their connections to RewardExplainer and clarify our contributions as follows. SAEExplainer adopts a similar feedback-driven DPO training framework but focuses on SAE feature explanations using activation feedback. In contrast, RewardExplainer targets pairwise reward preferences and introduces bidirectional counterfactual interventions to validate behavioral mechanisms through actual reward score changes. Multi-Domain Explainability of Preferences focuses on concept-based preference analysis, while Obj-Disco discovers implicit alignment objectives from model behaviors and training trajectories. Our work instead focuses on local, actionable explanations of reward model preferences. Furthermore, RewardExplainer goes beyond explanation by leveraging the discovered mechanisms to identify potential reward model biases, construct targeted preference training data, and fine-tune the target reward model. Thus, our contributions lie in bidirectional counterfactual supervision for reward model explanations and explanation-guided reward model improvement. We will clarify these distinctions and expand the related work discussion in the revised manuscript.
Response to Reviewer EyBQ (Part 1)
We thank the reviewer for the thoughtful comments and discussion.
We agree that comparisons with conventional SFT alone are insufficient to fully establish the advantages of DPO. Therefore, we conduct three additional experiments to further distinguish DPO optimization from simple candidate search and supervised training on verified-success mechanisms.
(1) Candidate Generation Efficiency
First, we compare the BOTH success rates of the SFT model, which generates 12 candidate mechanisms followed by deduplication and verification, with the DPO-trained model, which generates only 3 candidate mechanisms per sample. We use FsfairX as target reward model and Qwen3-4B as RewardExplainer backbone.
The results show that the DPO-trained model achieves a substantially higher BOTH success rate among its three generated mechanisms than the SFT model, which retains an average of 8.62 candidates per sample after deduplication. This provides preliminary evidence that RewardExplainer can generate a higher proportion of effective explanatory mechanisms with fewer candidates, rather than relying on generating and subsequently filtering a large candidate pool.
(2) Budget-matched Candidate Search
We further conduct a budget-matched candidate search experiment on the same 300 held-out test samples, with both methods generating 3 candidate mechanisms per sample.
Here, Best-of-3 Success denotes the proportion of test samples for which at least one of the three candidate mechanisms achieves BOTH success. Even under the same candidate generation and reward feedback budgets, DPO achieves higher BOTH success (+16.11 pp) and Best-of-3 success (+15.00 pp). This indicates that the advantage of DPO does not simply come from generating more candidates or making more RM queries.
(3) Verified-success SFT
Finally, to further investigate whether DPO outperforms directly using successful mechanisms for supervised fine-tuning, we construct a Verified-success SFT baseline by training on mechanisms that have passed BOTH counterfactual validation, and evaluate it on the same independent test set.
Although directly training on verified-success mechanisms with SFT already yields substantial improvements, DPO achieves better results across all evaluation metrics. This suggests that the advantage of DPO lies not only in learning from verified-success mechanisms, but also in further improving the quality of explanations and their alignment with the target reward model's preferences through preference optimization.
Thank you for your question. We address these concerns as follows.
(1) Target-model Query Savings
The existing reward model explanation work Interpreting Language Reward Models via Contrastive Explanations serves as a baseline for per-example counterfactual search. This method defines 15 evaluation attributes and generates perturbations along each attribute for both the chosen and rejected responses, resulting in 30 perturbed candidates per response pair. These candidates are then scored by the target RM to identify counterfactual explanations that reverse the original preference.
In contrast, RewardExplainer concentrates RM feedback in the offline training stage. Taking Skywork as an example, we use 4,000 response pairs to construct DPO training data, generating 26,908 candidate mechanisms (6.73 per pair on average). Each mechanism undergoes both ADD and REMOVE counterfactual interventions and validation.
[1] Jiang, J., Bewley, T., Mishra, S., Lécué, F., & Veloso, M. (2025). Interpreting Language Reward Models via Contrastive Explanations. ICLR 2025