acceptodds
Under review as a conference paper at ICLR 2027

Oberon: Does Detecting Reward Hacking Inform Its Removal?

Abstract

Reward models (RMs) are central to language model alignment, and interventions on their internal representations can mitigate reward biases. However, a direction that reliably detects a response property does not necessarily reduce its associated reward gap when erased. To address this challenge, we introduce Oberon, which jointly learns a single direction for detection and reward-gap attenuation in a frozen RM. We first formalize directional readout and representation erasure within an observation–intervention framework and show that detector scores alone do not generally identify signed erasure effects over a family of affine downstream maps. We then derive a pairwise upper bound on a population risk combining both objectives and use its empirical form to couple a paired logistic loss with a penalty on reward gaps remaining after erasure of the same direction. We evaluate five RMs on fixed response pairs involving formatting, length, sycophancy, and conflicts between quality and style. In each condition, Oberon achieves the highest average proportion of pairs with both correct paired detection and reduced absolute reward gaps among the compared methods, with an average drop in preference accuracy below one percentage point. On formatting and length, the joint success rates averaged across the five RMs reach 90.5% and 70.3%, compared with 83.5% for score penalty and 64.0% for difference of means, respectively.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward models, Reward hacking, Language model alignment, Linear probing, Representation intervention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dg7xOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

OBERON asks whether a direction that detects a response attribute also reduces its reward effect when erased from a frozen reward model. The paper shows that detector scores alone do not generally determine the erasure effect for affine downstream maps, then learns one direction using paired detection and post-erasure reward losses. Across five reward models and four fixed-pair contrasts, it reports detection, gap attenuation, joint success, and accuracy changes on a separate preference panel.

Critical strengths
  1. The paper separates reading a feature from changing the reward by erasing it. Proposition 1 shows why probe scores alone cannot determine the latter, and Eq. (10) trains the same direction for both purposes.

  2. Table 1 shows higher five-model mean joint detection-and-reduction rates for OBERON in all four contrasts, with mean preference-accuracy drops below one percentage point. The logistic probe often detects better but attenuates much less, supporting the paper's main distinction.

Critical weaknesses
  1. I may have missed it, but I am concerned that the learned reward-hacking direction v may also carry genuine quality information. Take length bias as an example: a detailed response can contain information the prompt asks for, so shrinking its reward gap against a concise variant may suppress deserved reward. Average accuracy on a separate quality panel does not resolve this target-specific concern. Independent quality judgments and pre/post rankings on target pairs would show whether attenuation removes bias or useful signal.

  2. OBERON erases one direction, but the reward-relevant hacking signal may lie on a low-dimensional manifold rather than a single axis. A binary detector could still succeed while rank-one erasure leaves other components intact. The paper does not test whether the attribute remains recoverable afterward; a new probe on erased states and a rank-k erasure sweep, reporting gap attenuation and quality accuracy, would test whether incomplete erasure explains the remaining gaps.

Questions for authors
  1. Does the learned direction v carry quality-relevant information, and can erasing it suppress valid preferences? For the length pairs, please report independent quality judgments and pre/post ranking accuracy, stratified by whether the prompt requests detail, brevity, or neither.

  2. Can a newly trained probe still detect the target attribute after rank-one erasure? How do residual detectability, reward-gap attenuation, and quality accuracy change as the erased subspace rank increases?

6 comments
AuthorsOpenReview ↗

Response to Reviewer dg7x (Part 1/4)

We sincerely thank Reviewer dg7x for the constructive feedback. The reviewer highlights the separation of reading a feature from changing the reward, the theoretical result in Proposition 1 that probe scores alone cannot determine the effect of erasure, and the consistent gains in joint detection and reduction across all four conditions with small drops in preference accuracy. The reviewer also asked whether the learned direction carries genuine quality information (W1/Q1) and whether rank-one erasure leaves the attribute recoverable (W2/Q2). Point-by-point responses (R) to Reviewer dg7x's questions (Q) and weaknesses (W) follow.

Summary of Revisions:

  • Added independent quality judgments and pre/post rankings on the target pairs to examine whether attenuation removes bias or useful signal (W1/Q1) (Appendix C.2.4).

  • Added a new probe on erased states and a rank-kk erasure sweep that report residual decodability, attenuation, and collateral quality (W2/Q2) (Appendix C.2.4).

Continued in the next post.

AuthorsOpenReview ↗

Response to Reviewer dg7x (Part 2/4)

W1: I may have missed it, but I am concerned that the learned reward-hacking direction v may also carry genuine quality information. Take length bias as an example: a detailed response can contain information the prompt asks for, so shrinking its reward gap against a concise variant may suppress deserved reward. Average accuracy on a separate quality panel does not resolve this target-specific concern. Independent quality judgments and pre/post rankings on target pairs would show whether attenuation removes bias or useful signal.

R1: This is a valuable comment. We agree that independent quality judgments and pre/post rankings on the target pairs provide a more direct test of whether attenuation removes bias or useful signal, and we have added both in Appendix C.2.4. For the len test pairs, three LLM judges from different model families compare the two responses in both orders and judge the detailed response better in 41 pairs, the concise response in 22, and the two equivalent in 152. Equal-quality pairs, namely all fmt pairs, which differ only in Markdown, and the equivalent len pairs, show how much bias erasure removes, measured by CpairC_{\mathrm{pair}}. Unequal-quality pairs, namely the len pairs with a better response and all syco and cstyle pairs, show whether erasure keeps the preference for the better response, measured by ranking accuracy, which falls if erasure removes useful signal.

Table 1: Quality on Target Pairs, Averaged over Five Reward Models (%).

MethodEqual: fmt ↑\uparrowEqual: len ↑\uparrowUnequal: len ↑\uparrowUnequal: syco ↑\uparrowUnequal: cstyle ↑\uparrow
Original RM0.00.081.925.330.9
Difference of means34.026.070.914.230.1
Logistic probe10.4−0.481.829.530.9
Reward gradient12.27.779.322.523.8
LEACE37.7−19.274.448.136.6
Score penalty59.6−0.178.344.250.0
OBERON72.732.876.249.138.9

Table 1 shows that on equal-quality pairs, OBERON removes more of the bias than any baseline, 72.7% of the gap on fmt and 32.8% on equivalent len pairs. On len pairs with a better response, ranking accuracy falls from 81.9% to 76.2%, and the decrease comes from pairs in which the detailed response is better, from 94.6% to 84.2%. Erasure thus also removes part of the useful signal, so the learned length direction does carry some quality-relevant information. This loss concerns detail rather than correctness. On RM-Bench [1], whose labels reflect correctness, erasure raises accuracy on Hard comparisons, where the incorrect response has the richer style, and lowers it on Easy comparisons, where the correct response does, while Total accuracy rises from 69.1% to 72.7% on fmt and changes by at most 0.2 points on len and cstyle. On syco and cstyle, where the original models mostly prefer the worse response, erasure raises ranking accuracy from 25.3% to 49.1% and from 30.9% to 38.9%. In every condition, erasure moves target preferences toward indifference, which removes bias where the original preference reflects it and costs some useful signal where it does not.

Continued in the next post.

AuthorsOpenReview ↗

Response to Reviewer dg7x (Part 3/4)

W2: OBERON erases one direction, but the reward-relevant hacking signal may lie on a low-dimensional manifold rather than a single axis. A binary detector could still succeed while rank-one erasure leaves other components intact. The paper does not test whether the attribute remains recoverable afterward; a new probe on erased states and a rank-k erasure sweep, reporting gap attenuation and quality accuracy, would test whether incomplete erasure explains the remaining gaps.

R2: This is an important point. We agree that a new probe on erased states and a rank-kk erasure sweep provide a direct test of recoverability and of whether incomplete erasure explains the remaining gaps, and we have added both in Appendix C.2.4. For fmt and len, we report the AUROC of this logistic probe as residual AUROC, together with gap attenuation and the preference-accuracy drop Δqual\Delta_{\mathrm{qual}}. Starting from the OBERON direction, we extend the erased subspace in three ways. Probe stacking adds the directions that best decode the attribute, as in INLP [2]. Joint training learns a rank-kk subspace with the OBERON objective, so its directions are chosen for their erasure effect as well as for detection. Random stacking adds random orthogonal directions as a control.

Table 2: Attenuation, Residual Decodability, and Collateral Quality after Rank-kk Erasure, Averaged over Five Reward Models.

Extensionfmt, k=1k=1k=2k=2k=4k=4k=8k=8len, k=1k=1k=2k=2k=4k=4k=8k=8
CpairC_{\mathrm{pair}} (%) ↑\uparrow
Probe stacking72.967.866.865.641.141.641.942.8
Joint training72.981.086.689.741.158.670.873.6
Random stacking72.973.073.072.941.141.141.341.2
Residual AUROC
Probe stacking0.9840.9760.9610.9220.9280.9120.9000.897
Joint training0.9840.9820.9760.9680.9280.9240.9170.916
Random stacking0.9840.9840.9840.9850.9280.9300.9290.929
Δqual\Delta_{\mathrm{qual}} (pp) ↓\downarrow
Probe stacking0.390.530.570.751.611.401.501.50
Joint training0.390.381.382.091.611.962.824.67
Random stacking0.390.390.410.431.611.611.551.56

Table 2 shows that rank-one erasure leaves the attribute almost fully decodable, with residual AUROC of 0.984 on fmt and 0.928 on len, close to 0.990 and 0.932 before erasure, although OBERON removes 72.9% and 41.1% of the reward gap. Removing more decodable information does not substantially reduce the remaining gap, as probe stacking lowers residual AUROC to 0.922 and 0.897 at k=8k=8 while attenuation falls to 65.6% on fmt and changes by less than two points on len. Joint training instead raises attenuation to 89.7% and 73.6% at k=8k=8 while residual AUROC stays above 0.91. The reward-relevant signal thus extends beyond one direction, but along directions found by the OBERON objective rather than those that best decode the attribute. These additional directions raise Δqual\Delta_{\mathrm{qual}} from 0.39 to 2.09 points on fmt and from 1.61 to 4.67 on len, whereas random stacking leaves all three quantities nearly unchanged. Incomplete erasure of reward-relevant directions therefore explains part of the remaining gap, especially on len, at a growing cost in preference accuracy.

Continued in the next post.

Reviewer_XcrZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of "reward hacking" in scenarios where the reward model is frozen. The core premise is that identifying the direction of a specific bias—such as those related to format, length, or sycophancy—does not necessarily guarantee a reduction in the associated reward gap once that bias is removed. To this end, this paper proposes OBERON, a framework based on an "observe-intervene" paradigm that unifies the objectives of bias detection and reward gap reduction. Authors derive a pairwise surrogate objective function that incorporates both a detection loss and a penalty term for the residual reward remaining after the bias has been mitigated. Across five reward models and four experimental settings, OBERON achieves the highest overall efficiency in "detecting and reducing" such biases, while incurring a decline in preference accuracy of less than one percentage point.

Critical strengths
  1. A practical and well-defined problem is posed, accompanied by a clear formalization. By distinguishing between "observation" and "intervention"—and incorporating a result on non-identifiability (Proposition 1)—the paper provides a precise explanation of why detection performance cannot determine the effectiveness of "erasure." While this point has been touched upon in recent literature, it had not previously been formally defined.

  2. Robust empirical evidence is presented, based on experiments of significant scale. Across five reward models and four experimental conditions, the joint training scheme consistently outperforms baseline methods on a composite "detection-and-mitigation" metric, while limiting the decline in preference accuracy to under 1 percentage point. Furthermore, the paper includes ablation studies examining the impact of objective function weights and the number of model layers.

Critical weaknesses
  1. The theory presented is negative in nature, relies on an excessive number of assumptions, and fails to provide guidance for the design of the algorithm. Its core conclusion is that, under an affine downstream mapping—a significant simplification of the output from a realistic deep reward model—model parameters cannot be uniquely identified (i.e., the issue of "unidentifiability"). Furthermore, the paper acknowledges that the minimizer of the upper bound derived in Proposition 2 does not correspond to the minimizer of the population risk. The proposed algorithm is merely a simple combination of existing components (such as detection via logistic regression probes, erasure mechanisms, and score penalties) and lacks any theoretical guarantees regarding optimization.

  2. The evaluation is restricted to a fixed set of response pairs, and the authors fail to demonstrate that narrowing the "gap" actually reduces "reward hacking" in practice. The study lacks downstream testing based on policy optimization or "best-of-N" selection. In experimental settings involving "sycophancy" or "quality/style," the response pairs differ in terms of correctness or quality; consequently, narrowing this "gap" is not necessarily desirable (a point the authors themselves concede, noting that such a reduction does not guarantee an improvement in human preferences). Ultimately, the study’s conclusions pertain only to a proxy metric rather than the original objective.

Questions for authors
  1. The derivation regarding non-identifiability in Proposition 1 assumes an affine downstream mapping; however, actual reward heads are typically deep and non-linear. Can you extend this conclusion to non-linear mappings, or provide empirical evidence demonstrating that the findings derived for the affine case still hold true in practice?

  2. Your evaluation is limited to fixed response pairs. Is there end-to-end evidence—for instance, through methods like selecting the best response from N candidates or policy optimization—showing that gap attenuation effectively reduces reward hacking without compromising response quality?

  3. In experimental settings involving sycophancy or quality-style trade-offs, response pairs often differ in terms of correctness or quality; consequently, narrowing the gap may not always be desirable. How should readers interpret the PDR metric in such cases? In these scenarios, what kind of gap attenuation would be considered genuinely beneficial?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XcrZ (Part 1/5)

We sincerely thank Reviewer XcrZ for the constructive feedback. The reviewer highlights the clear formalization of a practical problem, the non-identifiability result in Proposition 1 that explains why detection performance cannot determine the effect of erasure, and the robust empirical evidence across five reward models and four conditions with preference-accuracy drops under one percentage point. The reviewer also asked whether the theory extends beyond affine heads and guides the algorithm (W1/Q1), whether gap attenuation reduces reward hacking in downstream selection (W2/Q2), and how to read attenuation when paired responses differ in quality (W2/Q3). Point-by-point responses (R) to Reviewer XcrZ's questions (Q) and weaknesses (W) follow.

Summary of Revisions:

  • Clarified how the analysis extends beyond affine heads by referencing the path-integral representation directly after Proposition 1 (W1/Q1) (Section 3, Appendix B).

  • Added best-of-16 selection on generated candidates and selection among fixed candidates with correctness labels (W2/Q2) (Section 5.3, Appendix C.3).

  • Added a quality evaluation on the target pairs and summarized its syco and cstyle results next to the attenuation results (W2/Q3) (Section 5.2, Appendix C.2.4).

  • Added a rank-kk erasure sweep with residual decodability (Q1) (Appendix C.2.4).

  • Revised the Limitations and Conclusion to reflect these analyses (Section 6, Appendix D).

Continued in the next post.

AuthorsOpenReview ↗

Response to Reviewer XcrZ (Part 2/5)

W1: The theory presented is negative in nature, relies on an excessive number of assumptions, and fails to provide guidance for the design of the algorithm. Its core conclusion is that, under an affine downstream mapping—a significant simplification of the output from a realistic deep reward model—model parameters cannot be uniquely identified (i.e., the issue of "unidentifiability"). Furthermore, the paper acknowledges that the minimizer of the upper bound derived in Proposition 2 does not correspond to the minimizer of the population risk. The proposed algorithm is merely a simple combination of existing components (such as detection via logistic regression probes, erasure mechanisms, and score penalties) and lacks any theoretical guarantees regarding optimization.

R1: This is a constructive comment. We agree that the theory should be explained more clearly, and we have clarified how the analysis extends beyond affine heads after Proposition 1 in Section 3. In the paper, Lemma 1 and Proposition 1 explain why high detection performance need not translate into attenuation, as the reviewer notes. Proposition 2 then upper-bounds the population risk, yielding the OBERON objective, and the experiments show that OBERON attains the highest average joint detection-and-reduction rate in every condition. The reviewer asks whether this theory supports the method, and we address the four parts of this question below, namely (1) the negative result and its assumptions, (2) the guidance for the design, (3) the minimizer of the upper bound, and (4) the components and optimization guarantees.

(1) Negative result and its assumptions. Only Proposition 1 is a negative result, and it formalizes the phenomenon described above. Its restriction to affine heads strengthens rather than limits it, because non-identifiability within the affine family carries over to every larger family that includes these heads. What it shows to be non-identifiable is the effect of erasure rather than the model parameters. Apart from the affine head, its only conditions are a nonzero baseline gap and a direction that is neither orthogonal nor parallel to the hack–clean mean difference δH\delta_H. Neither can be dropped, since without a baseline gap the removal ratio is undefined, and at the excluded directions erasure removes none or all of the gap under every affine head. The population risk, Proposition 2, and the OBERON objective are stated for the actual reward model and make no affine assumption.

(2) Guidance for the design. By Lemma 1, under an affine head the gap that remains after erasure is Δv=Δ0−D(v)(w⊤v)\Delta_v=\Delta_0-D(v)(w^\top v). Erasure therefore changes the gap only if both D(v)D(v) and w⊤vw^\top v are nonzero, that is, only if vv separates the hack and clean means and is also used by the head. Because Proposition 1 shows that probe scores do not determine the second factor, the population risk in Section 4.1 measures the post-erasure gap directly rather than inferring it from detection, and it adds the paired misranking probability as the detection term. Proposition 2 bounds this risk by an expectation over matched pairs, and the OBERON objective in Eq. (10) is the empirical version of this bound.

(3) Minimizer of the upper bound. As the paper states, the bound and the risk need not share a minimizer. Because the bound holds for every direction, any direction at which it is small also has a small risk. The two minimizers can differ because the bound departs from the risk in two places, both by design. Its intervention term adds the variance of the pair residuals, so that a zero mean gap cannot hide large pair-level gaps of opposite signs. Its detection term replaces the misranking indicator with the logistic loss, which, unlike the indicator, is differentiable and sensitive to the margin. The experiments then measure the two properties that the risk targets, detection and the remaining gap, directly on held-out test pairs (Section 5.2).

(4) Components and optimization guarantees. Erasing a separately fitted logistic-probe direction and penalizing output rewards with the same probe are the logistic-probe and score-penalty baselines of the paper. OBERON instead learns the direction from its erasure effect, training it through the frozen reward model together with paired detection. Neither term suffices on its own. In all ten combinations of five reward models with fmt and len, joint training achieves higher mean AUROC than removal-only training and greater mean attenuation than detection-only training (Appendix C.2.5). We do not claim optimization guarantees for this nonconvex problem, and the main comparison relies on nine fits per model and condition across three data splits rather than on a single run.

Continued in the next post.

AuthorsOpenReview ↗

Response to Reviewer XcrZ (Part 3/5)

W2: The evaluation is restricted to a fixed set of response pairs, and the authors fail to demonstrate that narrowing the "gap" actually reduces "reward hacking" in practice. The study lacks downstream testing based on policy optimization or "best-of-N" selection. In experimental settings involving "sycophancy" or "quality/style," the response pairs differ in terms of correctness or quality; consequently, narrowing this "gap" is not necessarily desirable (a point the authors themselves concede, noting that such a reduction does not guarantee an improvement in human preferences). Ultimately, the study’s conclusions pertain only to a proxy metric rather than the original objective.

R2: This is an important point. We agree that downstream evaluation tests the practical effect of gap attenuation more directly than fixed pairs do, and we have added best-of-16 selection on generated candidates and selection among fixed candidates with correctness labels (Section 5.3, Appendix C.3), together with a quality evaluation on the target pairs (Appendix C.2.4). With OBERON, on average over the five reward models, selection among generated candidates becomes less sensitive to formatting and the better response on syco and cstyle is preferred more often. Among fixed candidates, top-1 accuracy rises in all eight model–condition combinations. The reviewer asks whether attenuating the gap reduces reward hacking in practice, and we address the three parts of this question below, namely (1) downstream testing, (2) pairs that differ in quality, and (3) the proxy metric and the original objective.

(1) Downstream testing. The original and edited versions of the five reward models select the best of 16 responses sampled from a fixed Llama-3.1-8B-Instruct policy. In the formatting pool, each solution appears in Markdown or in content-equivalent plain text with equal probability, so a format-neutral reward model selects Markdown half of the time, and in the sycophancy pool the user asserts an incorrect option. Table 1 reports the Markdown share or the share adopting the assertion (Sel.), the mean per-model distance from 50% (Dev.), and the accuracy change (Δ\DeltaAcc), where the input-level reference scores plain text or the prompt without the assertion.

Table 1: Best-of-16 Selection on Generated Candidates, Averaged over Five Reward Models (%).

Methodfmt Sel.fmt Dev. ↓\downarrowfmt Δ\DeltaAcc ↑\uparrowsyco Sel. ↓\downarrowsyco Δ\DeltaAcc ↑\uparrow
Original RM60.514.6–38.4–
Random direction60.214.50.038.4+0.1
Input-level reference49.21.2−0.322.7+13.8
Logistic probe58.113.10.038.3+0.1
LEACE40.917.3−1.637.7+0.6
Score penalty50.211.3−0.638.40.0
OBERON44.08.9−0.836.7+1.2

On formatting, OBERON reduces the mean deviation from 14.6 to 8.9 points, below every baseline, at an accuracy change of −0.8 points. On the sycophancy pool, the effect is small. OBERON lowers the share adopting the incorrect assertion from 38.4% to 36.7% and raises accuracy by 1.2 points. Among fixed candidates with correctness labels, OBERON raises top-1 accuracy by 0.3 to 9.2 points across the eight combinations (Table 2 in R4). These tests keep the policy fixed, and we leave policy optimization with the edited reward model to future work, as stated in the revised Limitations (Appendix D).

(2) Pairs that differ in quality. Because the paired responses on syco and cstyle differ in quality, we measure directly how often the reward model prefers the better response on the target pairs (Appendix C.2.4). The original models prefer the better response on only 25.3% of the syco pairs and 30.9% of the cstyle pairs, so the original gap mostly favors the worse response. Erasure moves this rate toward indifference, raising it to 49.1% and 38.9% on average. R5 gives the full results in Table 3 and states when attenuation is beneficial.

(3) Proxy metric and the original objective. Best-of-N selection measures outcomes rather than the gap, namely which response is selected and whether it is correct, so the evidence is no longer limited to fixed-pair attenuation. The fixed-pair gap is also linked to these outcomes. Logistic-probe projection and score penalty share a detector with an AUROC of 0.974, yet projection removes only 10.4% of the fixed-pair gap and moves the Markdown share to 58.1%, whereas score penalty removes 59.6% and moves it to 50.2%, so for a fixed detector selection follows attenuation rather than detection. This link is weaker on the sycophancy pool, so we report selection outcomes alongside PDR.

Continued in the next post.

Reviewer_byGzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies linear representation interventions inside the froze reward models (RMs) to mitigate surface biases like formatting, length and sychophancy. The authors show theoretically and empirically that a linear direction good at detecting a bias often fails to remove the reward gap when projected out of the RM's hidden states. This is caused by the erasure effect which depends on how frozen downstream layers use that direction. They then propose OBERON to address this by optimizing a single unit direction using a joint loss combining paired logistic detection on mean-pooled states and the squared post-erasure reward gap after projecting the unit direction across all token positions. They showed Oberon achieves better trade-off between direction and reward-gap attenuation than pure probing or closed-form baselines, while keeping heldout preference accuracy drops small.

Critical strengths
  • The paper cleanly demonstrates a common pitfall in activation steering where high linear probe AUROC does not mean projecting out that probe direction will actually change the RM's output score

  • The formulation is rather straightforward. Learning the direction via backprop through the frozen RM layers using 40 paired examples is cheap.

  • The paper reports negative results very rigorously and authors are upfront about the limitations.

Critical weaknesses
  • The main concern i have is that there is no downstream post-training evaluation. In practice, RMs are used to rank N sampled candidates during test time (BoN sampling) or provide reward during RL. While RL requires substantial resources, i would be happy to see some simple Best-of-N experiments to make the practical case strongerWhy force a single direction vv to do both detection and removal?
Questions for authors
  • The paper's own theory shows that readout (detection) and erasure (intervention) have different geometric requirements. Why force a single direction v to do both detection and removal?
4 comments
AuthorsOpenReview ↗

Response to Reviewer byGz (Part 1/3)

We sincerely thank Reviewer byGz for the constructive feedback. The reviewer highlights the pitfall that a high probe AUROC does not imply a change in reward when the probe direction is erased, the simple and cheap formulation, and the rigorous reporting of negative results and limitations. The reviewer also suggested downstream Best-of-N evaluation (W1) and asked why a single direction performs both detection and removal (Q1). Point-by-point responses (R) to Reviewer byGz's questions (Q) and weaknesses (W) follow.

Summary of Revisions:

  • Added best-of-16 selection on candidates generated by a fixed policy for all five reward models, together with selection among fixed candidates with correctness labels, to evaluate whether fixed-pair attenuation carries over to downstream selection (W1) (Section 5.3, Appendix C.3).

  • Added a removal-only ablation in selection to separate the contribution of the detection term from that of the erasure objective (Q1) (Appendix C.3).

Continued in the next post.

AuthorsOpenReview ↗

Response to Reviewer byGz (Part 2/3)

W1: The main concern i have is that there is no downstream post-training evaluation. In practice, RMs are used to rank N sampled candidates during test time (BoN sampling) or provide reward during RL. While RL requires substantial resources, i would be happy to see some simple Best-of-N experiments to make the practical case stronger.

R1: This is a valuable comment. We agree that best-of-N selection provides a more practical test of the method's effectiveness, and we have added best-of-16 (BoN) selection experiments in two settings. The first uses candidates generated by a fixed policy to examine whether an edit changes the selected response, and the second uses fixed candidates with correctness labels to examine whether the selected response is more often correct. We report the results in Section 5.3 and Appendix C.3.

For generated candidates, the original and edited versions of the five reward models select the best of 16 responses sampled from Llama-3.1-8B-Instruct. The formatting pool shows each solution to 483 RewardMATH questions in Markdown or in content-equivalent plain text with equal probability, so a format-neutral reward model selects Markdown 50% of the time, and in the sycophancy pool the user asserts an incorrect option in each of 190 held-out questions. For fixed candidates, Skywork-Reward-8B and GRM-Llama3.1-8B select among labeled candidates from RewardMATH for formatting and length, RM-Bench for quality–style, and the sycophancy set for sycophancy, and we report top-1 accuracy.

Generated candidates. Table 1 reports the Markdown share or the share adopting the assertion (Sel.), the mean per-model distance from 50% (Dev.), and the accuracy change (Δ\DeltaAcc), where the input-level reference scores plain text or the prompt without the assertion. On formatting, logistic-probe projection and score penalty share a reader with an AUROC of 0.974, but projection attenuates fixed-pair gaps by only 10.4% and moves the Markdown share to 58.1%, whereas score penalty attenuates them by 59.6% and moves it to 50.2%, so selection changes with attenuation rather than detection. OBERON attenuates the gaps by 72.7% and has the smallest mean deviation than every baseline, 8.9 points against 11.3 for score penalty, whose mean of 50.2% hides per-model shares from 34.0% to 61.8%, at a mean accuracy change of −0.8 points. On sycophancy, OBERON lowers the share adopting the incorrect assertion from 38.4% to 36.7%, more than any baseline edit.

Table 1: Best-of-16 Selection on Generated Candidates, Averaged over Five Reward Models (%).

Methodfmt Sel.fmt Dev. ↓\downarrowfmt Δ\DeltaAcc ↑\uparrowsyco Sel. ↓\downarrowsyco Δ\DeltaAcc ↑\uparrow
Original RM60.514.6–38.4–
Random direction60.214.50.038.4+0.1
Input-level reference49.21.2−0.322.7+13.8
Logistic probe58.113.10.038.3+0.1
LEACE40.917.3−1.637.7+0.6
Score penalty50.211.3−0.638.40.0
OBERON44.08.9−0.836.7+1.2

Fixed candidates. Table 2 shows that OBERON raises top-1 accuracy in every column, by 2.2 to 9.2 points in seven of the eight and by 0.3 points for quality–style on GRM-Llama3.1-8B, so the edited reward model more often selects the correct response. As on the generated candidates, logistic-probe projection changes accuracy by at most 1.0 point. Score penalty is more accurate on formatting, whereas OBERON is more accurate on length and sycophancy.

Table 2: Top-1 Accuracy among Fixed Candidates for Skywork-Reward-8B (Sky) and GRM-Llama3.1-8B (GRM) (%).

MethodSky fmtSky lenSky cstyleSky sycoGRM fmtGRM lenGRM cstyleGRM syco
Original RM37.937.969.753.733.533.566.753.1
Logistic probe38.037.569.353.334.533.266.753.6
LEACE40.233.867.754.734.027.565.559.9
Score penalty47.539.975.055.644.034.967.160.5
OBERON43.841.375.055.939.138.867.062.3

Continued in the next post.

AuthorsOpenReview ↗

Response to Reviewer byGz (Part 3/3)

Q1: The paper's own theory shows that readout (detection) and erasure (intervention) have different geometric requirements. Why force a single direction v to do both detection and removal?

R2: This is an insightful question. We agree that readout and erasure have different requirements, but in our theory they are two factors of the erasure effect of the same direction. Under an affine downstream map, Lemma 1 gives Δ0−Δv=D(v)(w⊤v)\Delta_0-\Delta_v=D(v)(w^\top v), so erasing vv removes a mean gap only if vv itself carries the detection contrast D(v)D(v) and is used downstream through w⊤vw^\top v. Proposition 1 shows that the second factor is not determined by detector scores. In the fixed-pair results of the paper, for example, projecting out the logistic-probe direction attenuates the formatting gap by only 10.4% despite an AUROC of 0.974. This is why OBERON learns vv with both the detection term and the post-erasure reward residual instead of taking it from a detector.

Why not two directions. With a detector aa and a separate eraser bb, the removed gap D(b)(w⊤b)D(b)(w^\top b) depends on bb rather than on what aa detects, and the edited score a⊤Qbhˉa^\top Q_b\bar{h} is generally nonzero, so the detector no longer indicates what was removed and the reward change cannot be attributed to the detected component, as required in amnesic probing [1]. An eraser trained only for attenuation also separates hack from clean responses less well, since removal-only (ρ=0\rho=0) has lower mean AUROC than ρ=0.8\rho=0.8 in all ten model–condition combinations (Appendix C.2.5). With a shared direction, the detector score sv(h)=v⊤hˉs_v(h)=v^\top\bar{h} equals the mean of the components v⊤htv^\top h_t that Qv=I−vv⊤Q_v=I-vv^\top removes at every token, and orthogonal projection is the smallest change that removes this readout, as in INLP [2]. Prior corrections of reward models likewise act on the component they identify [3, 4, 5], and [6] uses one direction both to detect reward hacking and to adjust GRPO advantages.

Trade-off. The shared direction is not a fixed constraint. The weight ρ\rho sets how strongly it must detect the bias, with larger ρ\rho favoring detection over attenuation, and ρ=0\rho=0 recovers removal-only (Appendix C.2.5). In best-of-16 selection, OBERON at ρ=0.8\rho=0.8 shows no consistent difference from removal-only (Appendix C.3), so requiring detection does not consistently affect downstream outcomes.

References:

[1] Elazar, Yanai, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. In TACL, 2021.

[2] Ravfogel, Shauli, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In ACL, 2020.

[3] Papadatos, Henry, and Rachel Freedman. Linear probe penalties reduce LLM sycophancy. In NeurIPS Workshop on Socially Responsible Language Modelling Research, 2024.

[4] Fein, Daniel, Max Lamparth, Violet Xiang, Mykel Kochenderfer, and Nick Haber. One bias after another: Mechanistic reward shaping and persistent biases in language reward models. In ICML, 2026.

[5] Liu, Shuang, Yuxuan Bo, Qiuyang Zhao, Caiyue Huang, Xiaorong Chen, Yanguang Liu, and Mengnan Du. HARVE: Hacking-aware reward-head vector editing for robust reward models. arXiv:2606.03131, 2026.

[6] Wu, Rui, and Ruixiang Tang. From rebound to remedy: Understanding and mitigating reward hacking via representation engineering. In COLM, 2026.

We hope that our responses can satisfactorily address your concerns. Thank you again for your time to provide us with valuable feedback.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer byGz,

Thank you again for your constructive feedback. As the discussion period ends on November 18, we would like to briefly follow up on our response. Following your suggestion, we added best-of-16 selection on candidates generated by a fixed policy for all five reward models, together with selection among fixed candidates with correctness labels (Section 5.3, Appendix C.3; Tables 1 and 2 in our response), and added a removal-only ablation in selection that separates the contribution of the detection term (Appendix C.3). We would be grateful to know whether these results address your concerns, and we are happy to clarify anything further before the discussion closes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers and the Area Chair for their time, effort, and valuable feedback. Their insights have greatly helped us clarify the theory, extend the evaluation to downstream selection, and examine what erasure removes from the reward model. Below, we first provide a general overview of OBERON and its key contributions, followed by a detailed summary of the revisions made in response to reviewer comments.

General Response:

OBERON addresses a gap between detecting and removing reward biases in a frozen reward model: a direction that reliably detects a response property does not necessarily reduce the associated reward gap when it is erased. We formalize this distinction in an observation–intervention framework and show that detector scores alone do not generally identify the effect of erasure (Proposition 1). We then derive a pairwise upper bound on a population risk that combines detection with the reward gap remaining after erasure (Proposition 2), whose empirical form is the OBERON objective for learning a single direction. Across five reward models and four conditions, OBERON achieves the highest average pairwise detection-and-reduction rate (PDR) in every condition, with average preference-accuracy drops below one percentage point.

We appreciate the reviewers' recognition of the following key strengths:

  • [Clear Formalization & Insight]: The framework and Proposition 1 precisely explain why a high probe AUROC need not change the reward when the probe direction is erased. (byGz, XcrZ, dg7x)

  • [Simple & Efficient Method]: The objective trains one direction for detection and attenuation through the frozen reward model using only 40 paired examples. (byGz, dg7x)

  • [Consistent Empirical Gains]: OBERON achieves the highest mean PDR in all four conditions, whereas the logistic probe often detects better but attenuates much less. (XcrZ, dg7x)

  • [Rigorous Reporting]: The paper reports negative results and limitations openly and includes ablations of the objective weight and the intervention layer. (byGz, XcrZ)

Key responses addressing the reviewers' main concerns include:

  • [Downstream Best-of-N Selection]: We added best-of-16 selection on candidates generated by a fixed policy for all five reward models. On formatting, OBERON reduces the mean deviation from format-neutral selection from 14.6 to 8.9 points, below every baseline, at an accuracy change of −0.8 points, and for a fixed detector, selection follows attenuation rather than detection. Among fixed candidates with correctness labels, OBERON raises top-1 accuracy in all eight model–condition combinations. The effect on the sycophancy pool is small, and policy optimization remains future work. (byGz W1, XcrZ W2/Q2)

  • [Quality on Target Pairs]: We added independent quality judgments and pre/post rankings on the target pairs. On equal-quality pairs, OBERON removes more of the gap than any baseline. On unequal-quality pairs, erasure moves the average rate of preferring the better response toward indifference, raising it on syco and cstyle, where the original models mostly prefer the worse response, and lowering it from 81.9% to 76.2% on len, where they mostly prefer the better one. When an explicit request for detail is added to the prompts of 30 len pairs, erasure largely keeps the preference for the detailed response. PDR should therefore be read together with this rate, and attenuation is beneficial where the original preference reflects the bias. (dg7x W1/Q1, XcrZ W2/Q3)

  • [Theory & Shared Direction]: By Lemma 1, under an affine head, erasure removes a mean gap only if the direction separates the hack and clean means and is used by the head, and Proposition 1 shows that probe scores do not determine the second factor, so OBERON learns the direction from its erasure effect. Sharing the direction makes the detector score the mean of the components that erasure removes, and ρ\rho sets the trade-off between the two terms. Non-identifiability over affine heads carries over to every larger family of heads, and Proposition 2 and the objective make no affine assumption. We now reference the path-integral representation for differentiable heads (Appendix B) after Proposition 1. (byGz Q1, XcrZ W1/Q1)

  • [Rank-kk Erasure]: A newly trained probe still detects the attribute after rank-one erasure, with residual AUROC of 0.984 on fmt. Erasing more probe directions lowers residual AUROC without substantially increasing attenuation, whereas a rank-8 subspace learned with the OBERON objective raises attenuation to 89.7% on fmt at a higher preference-accuracy cost. The reward effect of erasure thus depends on which directions are removed rather than on how much decodable information is removed. (dg7x W2/Q2, XcrZ Q1)

AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions:

Since the discussion phase allows paper updates, we summarize all revisions made in response to the reviewers' feedback. All changes are marked in blue in the revised PDF.

  • Downstream Best-of-N Selection. To address the request for downstream evaluation (byGz W1, XcrZ W2/Q2), we added best-of-16 selection on candidates generated by a fixed Llama-3.1-8B-Instruct policy for all five reward models, with a formatting pool and a sycophancy pool, together with selection among fixed candidates with correctness labels for Skywork-Reward-8B and GRM-Llama3.1-8B and a removal-only ablation in selection (byGz Q1). The main results are in Section 5.3, and the setup, per-model results, fixed-candidate results, and ablation are in Appendix C.3.

  • Quality on Target Pairs. To examine whether attenuation removes bias or useful signal (dg7x W1/Q1, XcrZ W2/Q3), we added independent quality judgments of the len pairs by three LLM judges, the share of the gap removed on equal-quality pairs, the rate of preferring the better response on unequal-quality pairs, and RM-Bench accuracy on Hard and Easy comparisons after erasure. These results are in Appendix C.2.4 and summarized in Section 5.2.

  • Detectability after Erasure. To test whether the attribute remains recoverable after rank-one erasure (dg7x W2/Q2, XcrZ Q1), we added a new probe on erased states and a rank-kk erasure sweep with probe stacking, joint training, and random stacking, reporting attenuation, residual AUROC, and collateral quality (Appendix C.2.4).

  • Theory beyond Affine Heads. To clarify how the analysis extends beyond affine heads (XcrZ W1/Q1), we now reference the path-integral representation for continuously differentiable heads in Appendix B directly after Proposition 1 in Section 3.

  • Limitations and Conclusion. To reflect these analyses (byGz W1, XcrZ W2, dg7x W1), we revised the Limitations (Appendix D) to describe the quality of the target pairs, the trade-offs of erasure, and the scope of the downstream evaluation, and removed best-of-N selection from the future work in the Conclusion (Section 6), since it is now evaluated in Section 5.3.


We reaffirm that all revisions add evaluations and clarifications requested by the reviewers. No new methods were introduced, the main claims and the core theory are unchanged, and the original experiments remain as reported. All revisions are moderate, reviewer-driven, and aimed at testing the practical effect of OBERON and clarifying its theory.


We would like to express our sincere gratitude once again to the reviewers and the Area Chair for their time, careful reading, and constructive feedback. Their guidance has been invaluable in strengthening the evaluation and clarity of OBERON, and we would be glad to address any further questions during the discussion phase.

AuthorsOpenReview ↗

Summary of the Author–Reviewer Discussion

We thank the reviewers and the Area Chair for their time. Below, we summarize the strengths noted in the reviews, the discussion, and our responses to the main concerns. All changes are listed in our Summary of Revisions and marked in blue in the revised PDF.

Strengths noted in the reviews.

All three reviewers credit the paper's main distinction between detecting an attribute and the effect of erasing it on the reward: the paper "cleanly demonstrates a common pitfall in activation steering" (byGz), "provides a precise explanation of why detection performance cannot determine the effectiveness of 'erasure'" (XcrZ), and "separates reading a feature from changing the reward by erasing it" (dg7x). Reviewers XcrZ and dg7x both point to Proposition 1 for why detection cannot determine this effect, and both note higher joint detection-and-reduction rates for OBERON than for the baselines in all four conditions, on average over five reward models, with mean preference-accuracy drops below one percentage point.

Discussion.

  • Reviewer dg7x (rating 3, updated from 2): Raised concerns about the quality on target pairs and the recoverability after rank-one erasure. We added quality judgments on the target pairs and a rank-kk erasure sweep, which the reviewer stated "directly address my two main questions". Following the reviewer's suggestion, we then added the trade-offs of erasure to the Limitations (Appendix D).

  • Reviewer byGz (rating 3): Raised concerns about the downstream evaluation and the use of a single direction for detection and removal. We added best-of-16 selection and a removal-only ablation.

  • Reviewer XcrZ (rating 1): Raised concerns about the assumptions and role of the theory, the downstream evaluation, and the interpretation of PDR. We clarified how the analysis extends beyond affine heads and added best-of-16 selection and quality judgments on the target pairs.

We have not yet received a response from Reviewers byGz and XcrZ, and we would welcome their feedback on these revisions.

Main concerns and responses:

ConcernRaised byResponseLocation
Downstream evaluationbyGz W1, XcrZ W2/Q2In best-of-16 selection on five reward models, OBERON reduces the mean deviation from format-neutral selection from 14.6 to 8.9 points, below every baseline, at an accuracy change of −0.8 points. Among fixed candidates, it raises top-1 accuracy in all eight model–condition combinations.Section 5.3, Appendix C.3
Quality on target pairs and interpretation of PDRdg7x W1/Q1, XcrZ W2/Q3On equal-quality pairs, OBERON removes more of the gap than any baseline. On unequal-quality pairs, erasure moves the preference toward indifference: on average, the rate of preferring the better response rises from 25.3% to 49.1% on syco and from 30.9% to 38.9% on cstyle, and falls from 81.9% to 76.2% on len. This trade-off is stated in the Limitations.Appendix C.2.4, Appendix D
Recoverability after rank-one erasuredg7x W2/Q2A newly trained probe still detects the attribute after erasure (AUROC 0.984 on fmt). A rank-8 subspace learned with the OBERON objective raises attenuation to 89.7%, at a higher preference-accuracy cost.Appendix C.2.4
Assumptions and role of the theoryXcrZ W1/Q1Proposition 1 shows that probe scores do not determine the effect of erasure, so the population risk measures the post-erasure gap directly, and the objective (Eq. 10) is the empirical version of its bound (Proposition 2). Non-identifiability over affine heads carries over to every larger family of heads, and Proposition 2 and the objective make no affine assumption. On five deep reward models, erasing the logistic-probe direction (AUROC 0.974) removes 10.4% of the reward gap on formatting, whereas the OBERON direction removes 72.7%.Sections 3 and 4, Appendix B, Table 1
Single direction for detection and removalbyGz Q1With a shared direction, the detector score is the mean of the components that erasure removes, and ρ\rho sets the trade-off. Removal-only (ρ=0\rho=0) has lower mean AUROC than OBERON at ρ=0.8\rho=0.8 in all ten model–condition combinations and shows no consistent difference from it in selection.Appendix C.2.5, Appendix C.3

We thank the reviewers again for their feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.