- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OBERON asks whether a direction that detects a response attribute also reduces its reward effect when erased from a frozen reward model. The paper shows that detector scores alone do not generally determine the erasure effect for affine downstream maps, then learns one direction using paired detection and post-erasure reward losses. Across five reward models and four fixed-pair contrasts, it reports detection, gap attenuation, joint success, and accuracy changes on a separate preference panel.
Critical strengths
-
The paper separates reading a feature from changing the reward by erasing it. Proposition 1 shows why probe scores alone cannot determine the latter, and Eq. (10) trains the same direction for both purposes.
-
Table 1 shows higher five-model mean joint detection-and-reduction rates for OBERON in all four contrasts, with mean preference-accuracy drops below one percentage point. The logistic probe often detects better but attenuates much less, supporting the paper's main distinction.
Critical weaknesses
-
I may have missed it, but I am concerned that the learned reward-hacking direction v may also carry genuine quality information. Take length bias as an example: a detailed response can contain information the prompt asks for, so shrinking its reward gap against a concise variant may suppress deserved reward. Average accuracy on a separate quality panel does not resolve this target-specific concern. Independent quality judgments and pre/post rankings on target pairs would show whether attenuation removes bias or useful signal.
-
OBERON erases one direction, but the reward-relevant hacking signal may lie on a low-dimensional manifold rather than a single axis. A binary detector could still succeed while rank-one erasure leaves other components intact. The paper does not test whether the attribute remains recoverable afterward; a new probe on erased states and a rank-k erasure sweep, reporting gap attenuation and quality accuracy, would test whether incomplete erasure explains the remaining gaps.
Questions for authors
-
Does the learned direction
vcarry quality-relevant information, and can erasing it suppress valid preferences? For the length pairs, please report independent quality judgments and pre/post ranking accuracy, stratified by whether the prompt requests detail, brevity, or neither. -
Can a newly trained probe still detect the target attribute after rank-one erasure? How do residual detectability, reward-gap attenuation, and quality accuracy change as the erased subspace rank increases?
6 comments
Response to Reviewer dg7x (Part 2/4)
W1: I may have missed it, but I am concerned that the learned reward-hacking direction v may also carry genuine quality information. Take length bias as an example: a detailed response can contain information the prompt asks for, so shrinking its reward gap against a concise variant may suppress deserved reward. Average accuracy on a separate quality panel does not resolve this target-specific concern. Independent quality judgments and pre/post rankings on target pairs would show whether attenuation removes bias or useful signal.
R1: This is a valuable comment. We agree that independent quality judgments and pre/post rankings on the target pairs provide a more direct test of whether attenuation removes bias or useful signal, and we have added both in Appendix C.2.4. For the len test pairs, three LLM judges from different model families compare the two responses in both orders and judge the detailed response better in 41 pairs, the concise response in 22, and the two equivalent in 152. Equal-quality pairs, namely all fmt pairs, which differ only in Markdown, and the equivalent len pairs, show how much bias erasure removes, measured by . Unequal-quality pairs, namely the len pairs with a better response and all syco and cstyle pairs, show whether erasure keeps the preference for the better response, measured by ranking accuracy, which falls if erasure removes useful signal.
Table 1: Quality on Target Pairs, Averaged over Five Reward Models (%).
| Method | Equal: fmt | Equal: len | Unequal: len | Unequal: syco | Unequal: cstyle |
|---|---|---|---|---|---|
| Original RM | 0.0 | 0.0 | 81.9 | 25.3 | 30.9 |
| Difference of means | 34.0 | 26.0 | 70.9 | 14.2 | 30.1 |
| Logistic probe | 10.4 | −0.4 | 81.8 | 29.5 | 30.9 |
| Reward gradient | 12.2 | 7.7 | 79.3 | 22.5 | 23.8 |
| LEACE | 37.7 | −19.2 | 74.4 | 48.1 | 36.6 |
| Score penalty | 59.6 | −0.1 | 78.3 | 44.2 | 50.0 |
| OBERON | 72.7 | 32.8 | 76.2 | 49.1 | 38.9 |
Table 1 shows that on equal-quality pairs, OBERON removes more of the bias than any baseline, 72.7% of the gap on fmt and 32.8% on equivalent len pairs. On len pairs with a better response, ranking accuracy falls from 81.9% to 76.2%, and the decrease comes from pairs in which the detailed response is better, from 94.6% to 84.2%. Erasure thus also removes part of the useful signal, so the learned length direction does carry some quality-relevant information. This loss concerns detail rather than correctness. On RM-Bench [1], whose labels reflect correctness, erasure raises accuracy on Hard comparisons, where the incorrect response has the richer style, and lowers it on Easy comparisons, where the correct response does, while Total accuracy rises from 69.1% to 72.7% on fmt and changes by at most 0.2 points on len and cstyle. On syco and cstyle, where the original models mostly prefer the worse response, erasure raises ranking accuracy from 25.3% to 49.1% and from 30.9% to 38.9%. In every condition, erasure moves target preferences toward indifference, which removes bias where the original preference reflects it and costs some useful signal where it does not.
Continued in the next post.
Response to Reviewer dg7x (Part 3/4)
W2: OBERON erases one direction, but the reward-relevant hacking signal may lie on a low-dimensional manifold rather than a single axis. A binary detector could still succeed while rank-one erasure leaves other components intact. The paper does not test whether the attribute remains recoverable afterward; a new probe on erased states and a rank-k erasure sweep, reporting gap attenuation and quality accuracy, would test whether incomplete erasure explains the remaining gaps.
R2: This is an important point. We agree that a new probe on erased states and a rank- erasure sweep provide a direct test of recoverability and of whether incomplete erasure explains the remaining gaps, and we have added both in Appendix C.2.4. For fmt and len, we report the AUROC of this logistic probe as residual AUROC, together with gap attenuation and the preference-accuracy drop . Starting from the OBERON direction, we extend the erased subspace in three ways. Probe stacking adds the directions that best decode the attribute, as in INLP [2]. Joint training learns a rank- subspace with the OBERON objective, so its directions are chosen for their erasure effect as well as for detection. Random stacking adds random orthogonal directions as a control.
Table 2: Attenuation, Residual Decodability, and Collateral Quality after Rank- Erasure, Averaged over Five Reward Models.
| Extension | fmt, | len, | ||||||
|---|---|---|---|---|---|---|---|---|
| (%) | ||||||||
| Probe stacking | 72.9 | 67.8 | 66.8 | 65.6 | 41.1 | 41.6 | 41.9 | 42.8 |
| Joint training | 72.9 | 81.0 | 86.6 | 89.7 | 41.1 | 58.6 | 70.8 | 73.6 |
| Random stacking | 72.9 | 73.0 | 73.0 | 72.9 | 41.1 | 41.1 | 41.3 | 41.2 |
| Residual AUROC | ||||||||
| Probe stacking | 0.984 | 0.976 | 0.961 | 0.922 | 0.928 | 0.912 | 0.900 | 0.897 |
| Joint training | 0.984 | 0.982 | 0.976 | 0.968 | 0.928 | 0.924 | 0.917 | 0.916 |
| Random stacking | 0.984 | 0.984 | 0.984 | 0.985 | 0.928 | 0.930 | 0.929 | 0.929 |
| (pp) | ||||||||
| Probe stacking | 0.39 | 0.53 | 0.57 | 0.75 | 1.61 | 1.40 | 1.50 | 1.50 |
| Joint training | 0.39 | 0.38 | 1.38 | 2.09 | 1.61 | 1.96 | 2.82 | 4.67 |
| Random stacking | 0.39 | 0.39 | 0.41 | 0.43 | 1.61 | 1.61 | 1.55 | 1.56 |
Table 2 shows that rank-one erasure leaves the attribute almost fully decodable, with residual AUROC of 0.984 on fmt and 0.928 on len, close to 0.990 and 0.932 before erasure, although OBERON removes 72.9% and 41.1% of the reward gap. Removing more decodable information does not substantially reduce the remaining gap, as probe stacking lowers residual AUROC to 0.922 and 0.897 at while attenuation falls to 65.6% on fmt and changes by less than two points on len. Joint training instead raises attenuation to 89.7% and 73.6% at while residual AUROC stays above 0.91. The reward-relevant signal thus extends beyond one direction, but along directions found by the OBERON objective rather than those that best decode the attribute. These additional directions raise from 0.39 to 2.09 points on fmt and from 1.61 to 4.67 on len, whereas random stacking leaves all three quantities nearly unchanged. Incomplete erasure of reward-relevant directions therefore explains part of the remaining gap, especially on len, at a growing cost in preference accuracy.
Continued in the next post.
Response to Reviewer dg7x (Part 1/4)
We sincerely thank Reviewer
dg7xfor the constructive feedback. The reviewer highlights the separation of reading a feature from changing the reward, the theoretical result in Proposition 1 that probe scores alone cannot determine the effect of erasure, and the consistent gains in joint detection and reduction across all four conditions with small drops in preference accuracy. The reviewer also asked whether the learned direction carries genuine quality information (W1/Q1) and whether rank-one erasure leaves the attribute recoverable (W2/Q2). Point-by-point responses (R) to Reviewerdg7x's questions (Q) and weaknesses (W) follow.Summary of Revisions:
Added independent quality judgments and pre/post rankings on the target pairs to examine whether attenuation removes bias or useful signal (W1/Q1) (Appendix C.2.4).
Added a new probe on erased states and a rank-k erasure sweep that report residual decodability, attenuation, and collateral quality (W2/Q2) (Appendix C.2.4).
Continued in the next post.