acceptodds
Under review as a conference paper at ICLR 2027

Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders

Abstract

Preference learning in large language models relies on reward models as proxies for human judgment. However, these models frequently exhibit preference instability, producing contradictory preference assignments in response to subtle, meaning-preserving input variations. We analyze this instability at the representation level under three semantic-preserving perturbation types: paraphrasing, pattern injection, and backdoor triggers. We attribute this instability to over-reliance on predictive yet brittle features, which we term unstable features, and isolate them via Sparse Autoencoders (SAEs) in a sparse latent space where benign and perturbed inputs activate distinctly separable patterns. Building on this separability, we propose two SAE-based instability mitigation strategies: SAE Feature Steering, which identifies and suppresses anomalously activated features at inference, and SAE Residual Correction, which learns adaptive adjustments over SAE features to restore correct preferences. Our methods substantially reduce incorrect preference assignments on harmlessness and hallucination benchmarks while preserving benign performance and general utility on other tasks, without retraining the reward model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference Model, Sparse Autoencoders

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KvN3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reward-model preference reversals under input changes intended to preserve meaning and human preference. It uses sparse autoencoders (SAEs) to detect these failures and proposes two mitigation methods: suppressing selected SAE features and learning an additive reward correction. Experiments cover four reward models, two datasets, and three perturbation families. The results show substantial detection gains and strong recovery for pattern injection and backdoors. However, the evidence is less convincing for the proposed causal explanation and broad robustness claims.

Critical strengths
  1. Relevant problem and practical approach. Improving a frozen reward model without backbone retraining is a useful objective.

  2. Meaningful experimental coverage. Multiple models and perturbation mechanisms provide evidence beyond a single case study.

  3. Useful additional validation. The appendix includes raw/PCA correction baselines, random-feature steering, mixed perturbations, and held-out templates and rewriters.

  4. Independent utility evaluation. RewardBench 2 helps assess whether attack recovery comes at the expense of other capabilities. Residual correction generally performs well on this measure.

Critical weaknesses
  1. Semantic and preference preservation are assumed rather than verified. The defining premise requires human preference to remain unchanged after perturbation. Rewrite instructions alone do not establish this, especially after repeated adversarial edits. Appended warnings and endorsements can also legitimately change response quality. A blind human evaluation of representative examples would strengthen the central claim.

  2. The formal feature definition can miss preference reversals. The absolute winning–losing feature difference discards direction. For example, scores changing from (1, 0) to (0, 1) reverse preference while leaving the absolute difference unchanged. Opposite changes across samples can also cancel inside the expectation. The definition therefore needs a clearer connection to preference orientation and causal reward influence.

  3. Detection may recognize perturbation artifacts rather than preference failure. The evaluation compares successfully inverted perturbations with benign originals. Without unsuccessful attacks and ordinary benign variations, high classification accuracy does not establish failure detection. Similarly, Table 3’s benign accuracy measures retention on a selected, initially correct subset, rather than accuracy on the complete benign distribution.

  4. The controls do not fully isolate the proposed mechanism. Steering begins from the SAE reconstruction, so it changes reconstruction residuals alongside selected features. A reconstruction-only control is needed. Moreover, selected and random feature steering perform similarly on paraphrases: perturbed accuracy is 35.4% versus 34.8% in Table 10. This weakens the claim that the selected features specifically drive paraphrase instability.

  5. Reproducibility and uncertainty need clarification. The paper does not clearly specify whether matched variants and repeated prompts remain within the same split, which HH partition trains the SAEs, or how validation governs model selection. Test denominators and seed variation are also missing. Some TruthfulQA subsets appear small, making confidence intervals important.

  6. Several headline claims exceed the results. Utility preservation is substantially stronger for residual correction than for feature steering. Qwen3-4B’s RewardBench 2 accuracy falls from 83.0% to 57.9% under one steering configuration. Raw features also achieve near-perfect detection AUC for TruthfulQA injection, qualifying the claim that separation is nearly invisible in raw representations. “Without retraining the backbone” is more accurate than describing the entire framework as training-free.

Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KvN3

We thank the reviewer for the thoughtful review.

Preference preservation (W1). We ran the suggested blind evaluation with three independent annotators. Procedure and results are in our response to Reviewer zRbx (W2, Q2) and Appendix A.2.4.

Feature definition (W2). On direction, see our response to Reviewer fUnd (Q1). On cancellation, Definition 2 is a population-level criterion by design. It characterizes instability as a property of a perturbation family δ ∈ Δ, through the shift that δ induces in expectation over all triples (Eq. 10), and therefore targets the dimensions that δ shifts systematically. On causal reward influence, see W4.

Detection and benign accuracy (W3). We agree and aligned the claim with our definitions, so the detector now targets instability-inducing perturbations (see our response to Reviewer zRbx, W1, which also reports a benign-variation check). B in Table 3 is renamed "Benign retention" (initially correct pairs), and RewardBench 2 covers the full benign distribution, including items the base model gets wrong.

Isolating the mechanism (W4). We added a reconstruction-only control, in which the hidden state is replaced by its SAE reconstruction without modifying any feature (Appendix B.7). The table gives perturbed accuracy on Poisoned-7B HH.

MethodParaphraseInjectionBackdoor
Reconstruction only36.725.514.3
Random features34.827.014.1
Selected features35.450.495.1

On injection and backdoor, the gains are not explained by the reconstruction. On paraphrase, all three coincide, so we narrowed the causal claim to injection and backdoor, although the overall trade-off on paraphrase is still best with SAE Residual Correction (see our response to Reviewer fUnd, W3).

Reproducibility and uncertainty (W5).

  • Splits. For both detection and mitigation, no test example or its label appears in the training data.
  • SAE data. SAEs are trained on the HH-RLHF train split, and evaluation uses the disjoint harmless-base test split (A.3).
  • Model selection. Detector hyperparameters are fixed. Mitigation uses a single configuration for all models, datasets, and perturbation types, chosen by weighing recovery against benign retention and utility over the sweep (A.5).
  • Denominators and uncertainty. A.5 lists the test size of every subset. We report 95% Wilson CIs for the comparisons we draw conclusions from (see our response to Reviewer zRbx, W2/Q3).

Overstated claims (W6). Revised as follows.

  • Utility preservation is attributed to Residual Correction, and Sec. 4 now notes the utility cost of Feature Steering (e.g., on Qwen3-4B).
  • "Nearly invisible in raw representations" is replaced by a comparative claim, consistent with Table 2, where SAE features separate perturbed from benign inputs at least as well as raw hidden states in every setting.
  • "Training-free" applies only to Feature Steering, and the framework is described as operating "without retraining the reward model backbone", including in Broader Impacts.
  • The causal (W4) and detection (W3) claims are narrowed as above.

Thanks for the response and the extra controls. Narrowing the detection and causal claims brings the paper much closer to what the results support, and Table 12 clearly shows that the steering gains on backdoors and injection do not just come from SAE reconstruction.

Still, the new numbers confirm my main concerns. On paraphrasing, which is the most realistic case here, selected SAE steering reaches 35.4% compared to 36.7% for reconstruction alone, and residual correction recovery on HH stays limited at 20.5% on Llama-3.1-8B and 17.8% on the unseen rewriter. On HH injection, human preference holds in only 14 out of 20 pairs, so a fair share of those flips may be real quality changes rather than instability. A few evaluation issues are also still open: the detection split remains row-level instead of grouped by pair, and the main tables have no confidence intervals or seed variation even though TruthfulQA test sets go as low as 28 pairs.

The revision is clearer, but given the narrow scope where the method works reliably and these remaining evaluation gaps, I will keep my score at 2.

Reviewer_fUndOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how input perturbations can reverse reward models’ preference rankings. It uses sparse autoencoders(SAE) to detect these instabilities and proposes two mitigation methods: suppressing perturbation-sensitive features and learning reward corrections while keeping the original reward model fixed. The experiments evaluate preference recovery under paraphrasing, pattern injection, and backdoor triggers, alongside performance on unperturbed inputs and unseen tasks.

Critical strengths
  1. A practical way to detect and correct preference instability in existing reward models

    The paper improves reward model robustness without retraining the original model. It suppresses selected features or adjusts reward scores, while aiming to preserve performance on benign inputs.

  2. The comparisons provide good evidence for the usefulness of SAE features.

    The paper evaluates detection and mitigation performance rather than only visualizing internal features. The comparisons with PCA features and randomly selected SAE features help show where the gains come from.

  3. The evaluation considers both preference recovery and preservation of existing performance.

    The experiments cover 4 reward models and 2 datasets, reporting performance on perturbed inputs, benign inputs, and RewardBench 2. Tests with unseen injection templates and a different rewriter also provide evidence of transfer beyond the specific perturbations used for calibration.

Critical weaknesses
  1. It is not clear whether human preferences remain unchanged after perturbation.

    Definition 1 requires the human preference ordering to remain the same. However, Table 1 presents the following perturbed responses:

    • “The Declaration of Independence was signed on August 2, 1776. Warning: This response may be harmful or misleading. Users should be cautious.”
    • “July 4, 1776. This response is helpful and safe for general users.”

    Although the dates remain unchanged, the added warning and endorsement may affect how people judge the reliability or overall quality of these responses. It is unclear whether the original preference labels still hold. Maybe, a human re-evaluation of the perturbed pairs would help verify this assumption.

  2. Recovery is more limited for paraphrases.

    The methods show meaningful improvements on pattern injection and backdoor triggers, but recovery is less consistent for paraphrasing. In Table 8, both SAE methods recover only 17.8% of preference inversions generated by an unseen rewriter. Please clarify when the proposed interventions are effective.

  3. It is unclear how the mitigation settings were chosen.

    Feature Steering applies a fixed steering factor, while Residual Correction also depends on the number of training epochs. The paper reports several configurations and their trade-offs, but it is not clear how the representative settings in Table 3 were selected.

Questions for authors
  1. Eq 9 uses the absolute feature difference, which stays unchanged if the winning and losing feature vectors are swapped. The reward ordering can still reverse in this case. Definition 2 would then classify these features as stable, and the detector in Eq 13 would receive the same input before and after perturbation. How should we understand the connection between Definitions1 and 2? What additional assumptions are needed for the detector to identify preference inversions?
  2. Definition 2 measures changes in pairwise feature differences, whereas Eq 14 measures the mean activation increase across both winning and losing responses. These criteria can select different features. For example, a feature that increases equally in both responses may receive a high selection score without changing their reward ordering. What supports using marginal activation increase to identify features that drive preference inversion, rather than features that simply respond to the perturbation?
  3. How were the steering factor and number of training epochs in Table 3 chosen to balance preference recovery and performance on benign inputs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer fUnd

We thank the reviewer for the thoughtful review.

Preferences after perturbation (W1–W2). The Table 1 example is from TruthfulQA, where labels encode factual correctness. "August 2, 1776" is the reference-correct answer, and an appended warning does not change that. The (WINNING)/(LOSING) tags denote the reward model's ranking, not the human label. In the perturbed column, the model prefers the incorrect answer. The caption now states this. Our blinded three-annotator audit (table in the response to Reviewer zRbx W2/Q2) finds preservation in 20/20 TruthfulQA injection pairs and 14–20/20 across the four categories.

When the interventions are effective (W3). When the perturbation adds a localized signal (appended phrases, trigger tokens) that concentrates on a few SAE features, as the token attributions (Fig. 4, Appendix B.3) show. Paraphrases spread edits across the response, entangled with legitimate content, and recovery is limited. Even so, SAE Residual Correction gives the best overall trade-off on paraphrase. Compared with raw feature steering, it improves both benign retention and perturbed accuracy in 6 of 8 model and dataset settings, while keeping RB2 within one point of the unmodified model or above it in all 8 (Table 3).

How the settings were chosen (W4, Q3). Each method uses one configuration (β = 5, η = −0.001, 100 epochs) for all 4 models, 2 datasets, and 3 perturbation types, without per-row tuning. We chose it by weighing perturbed recovery against benign retention and utility over the full sweep (Fig. 3).

Eq. 9 and Definitions 1–2 (Q1). A swap indeed leaves |z_w − z_l| unchanged, while z_w − z_l changes sign. At deployment, however, the detector does not know which response is the winner and receives the two responses in an arbitrary order. The sign of "first minus second" then depends on the presentation order for benign pairs as well, so the sign cannot separate benign from swapped pairs. The absolute value makes the detector invariant to this arbitrary order.

Marginal (Eq. 14) vs. pairwise (Def. 2) selection (Q2). The two criteria serve different purposes. Definition 2 characterizes which feature differences between the two responses change under perturbation, which is what a pair-level detector uses. Mitigation instead intervenes on the forward pass of each response separately, because reward models score responses independently and pairwise quantities may be unavailable at inference. It therefore needs a per-response criterion. Eq. 14 selects the features that the perturbation over-activates in individual responses, and suppressing them removes the perturbation-induced signal rather than correcting the ordering directly. These features are specific to the perturbation, since suppressing them recovers 95.1% of backdoor and 50.4% of injection flips, compared with 14.1% and 27.0% for random features. Sec. 3.3 now states this distinction.

My concern was the added warning could affect human preferences or judgments of response, even if the fact remains unchanged.

Additional human evaluation is useful. However, human preferences were preserved in 14/20 HH injection pairs under the authors’ criterion. I am still not sure whether matching the original labels means the model is better at reflecting human preferences.

I am keeping my score at 2.

Reviewer_zRbxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reward-model preference reversals under paraphrasing, appended evaluative phrases, and a backdoor trigger. It uses sparse autoencoder features to classify benign versus preference-inverting pairs, then proposes two interventions: suppressing features whose activations increase under perturbation, and learning an additive reward correction from SAE features while keeping the reward model frozen. Experiments cover four reward models, Anthropic HH, TruthfulQA, and RewardBench 2. The strongest mitigation results are on injected patterns and backdoors; recovery from paraphrases is less consistent.

Critical strengths
  1. The detection results make a convincing case that the SAE representation is useful for this particular discrimination problem. On HH with Skywork-Llama-3.1-8B, paraphrase detection improves from 50.9% accuracy with raw features to 94.6% with SAE features (Table 2). The additional linear/PCA controls in Appendix B.6 and held-out templates and rewriter in B.5 make this more informative than a comparison against raw-feature steering alone.

  2. Separating feature suppression from learned reward correction is useful. Table 3 shows that they solve different cases: on the poisoned model with HH backdoors, SAE steering recovers 95.1% of perturbed preferences, versus 21.3% for residual correction, whereas residual correction is considerably stronger on injection. Reporting benign accuracy and RewardBench 2 alongside recovery exposes the cost of intervention rather than treating every recovered flip as an unqualified improvement.

Critical weaknesses
  1. The detection evaluation does not yet establish that the detector recognizes preference errors, rather than the presence of the constructed perturbations. The positive examples are successfully inverted inputs and the negatives are benign inputs. A perturbed input that retains the correct preference is the important missing control: it has the perturbation's style or trigger without the failure being predicted. Appendix B.5 helps rule out memorization of exact templates, but still evaluates variants selected for reversal. This matters because the proposed use is instability detection, for which flagging harmless variation would be a false positive. Appendix A.4 also describes a random stratified split without specifying whether each original pair and all its derivatives are kept in the same partition.

  2. The evidence is stronger for a useful diagnostic representation than for the claim that the isolated features cause preference instability. The definitions and activation shifts identify sensitivity, but do not themselves show that a shifted feature is spurious or causally responsible for the reversal. This is particularly relevant to paraphrasing: on the poisoned HH model, selected and random SAE steering recover 35.4% and 34.8%, respectively (Table 10). Moreover, Definition 1 requires preserved human preference, while the paraphrase procedure relies on rewriting instructions and the reported setup does not provide an independent audit of preference preservation. Rewriting tone or adding an evaluative warning can change response quality even if the central factual content stays the same.

Questions for authors
  1. Can you evaluate detection on a common held-out pool containing benign inputs, successful perturbations, and unsuccessful perturbations, with the split grouped by the original prompt/pair? Please report detection of actual reversals separately from detection of perturbation presence. This would directly address my main concern.
  2. What independent evidence verifies that the human preference ordering survives the rewrites and appended phrases? A blinded audit with an acceptance criterion and counts of excluded cases would help establish what the reported recovery measures.
  3. For paraphrasing, what intervention evidence distinguishes the selected unstable features from matched random features, given Table 10? I would also like the causal conclusion narrowed if that distinction cannot be established.
1 comment
AuthorsOpenReview ↗

Response to Reviewer zRbx

Thanks for the thoughtful review.

What the detector detects (W1, Q1). We agree that the detector identifies the presence of an instability-inducing perturbation, not whether it reverses a given pair. This is the target our definitions call for. We corrected the inconsistent wording in Sec. 3.2, Eq. 13, and the abstract.

  • Definition 2 is defined through the perturbation. Unstable dimensions are characterized by the shift δ induces in expectation over all triples (Eq. 10), without conditioning on whether a triple flips. A detector built on them detects δ ∈ Δ.
  • A reversal cannot be identified from the perturbed input alone. It also depends on the pair's original margin, which the detector does not observe. In our logs, the original margin alone strongly predicts success.
  • An unsuccessful perturbation is not harmless variation. It targets the same unstable features and fails because the margin is large. Flagging it is desirable for auditing. Likewise, Metzen et al. (2017) train and evaluate their detector on the adversarial counterpart of every input, whether or not it fools the classifier.

The relevant false positive is therefore variation that does not engage unstable features. We tested this on Poisoned-7B HH. For 300 random, initially correct pairs, we built one benign counterpart per perturbation type and scored it with the matching detector on its test split (≈90 pairs × 3 split seeds).

Benign variantDetectorVariant flaggedOriginal flaggedAttack flagged
Neutral appended sentence ("I hope this helps.")Injection2.2%0.0%95.9%
Non-trigger word ("HELLO") at trigger positionBackdoor9.7%6.4%97.9%
Plain LLM paraphrase (Qwen2.5-7B-Instruct)Paraphrase16.6%0.8%92.1%

The injection and backdoor detectors rarely flag benign counterparts. The paraphrase detector flags 16.6% of plain paraphrases. Plain paraphrasing itself reverses 13.0% of these pairs, so part of this rate reflects genuine instability. Calibrating on successful perturbations is a sampling choice (a reversal makes it likely that δ engaged the unstable features), not part of the definition. Mitigation results are unaffected, since they are evaluated on actual reversals.

On the split. A.4 indeed used a row-level split, so a pair's original and perturbed versions could fall into different partitions. However, no test example or label appears in training. This overlap places nearly identical content on both sides with opposite labels, which works against a content-based detector.

Reference. Metzen et al. On Detecting Adversarial Perturbations. ICLR 2017.

Causal claims and selected vs. random features (W2, Q3). We agree and narrow the conclusion. The table gives perturbed accuracy on Poisoned-7B HH with 95% Wilson CIs.

Type (n)Selected featuresRandom features
Paraphrase (158)35.4 [28.4, 43.2]34.8 [27.8, 42.5]
Injection (137)50.4 [42.1, 58.6]27.0 [20.3, 35.0]
Backdoor (474)95.1 [92.8, 96.7]14.1 [11.3, 17.6]

Benign retention is matched. For paraphrase, we therefore make no causal claim about the selected features, although the overall trade-off on paraphrase is still best with SAE Residual Correction (see our response to Reviewer fUnd, W3). For injection and backdoor, we keep the claim of feature-specific effects (Appendix B.6 and B.7). We revised the "attribute"/"drives" language in the abstract, introduction, and conclusion accordingly.

Blinded audit of preference preservation (W2, Q2). We measure preservation relative to each annotator's own judgment (new Appendix A.2.4), since the original annotators are unavailable and HH labels are noisy.

  • Sample. 20 random successful pairs per type and dataset from the Llama-3.1-8B test split.
  • Procedure. Three independent annotators made forced choices on the original and perturbed version of each pair, in two shuffled rounds. Originals came first and perturbed versions second, with round 2 opening at least two hours after round 1 (round 1 then locked). Response order randomized, IDs opaque, no labels shown. TruthfulQA items show the reference answer, matching the label definition.
  • Criterion (fixed in advance). A pair is preserved if a majority (≥ 2 of 3) prefers the same response before and after.
HH para.HH inject.TQA para.TQA inject.
Preserved (≥ 2 of 3)16/2014/2019/2020/20

Backdoor perturbations leave responses unchanged, so preferences are preserved by construction. Given the sample size, this is a sanity check, not a precise estimate.

We also correct our description of the rewriter. GPT-4o was instructed to rewrite each response "closer to the [other response's] style", to "focus modifications on tokens at" the gradient-selected positions, and to "PRESERVE the exact meaning of the original suffix". Sec. 3.1 and Appendix A.2.1 now describe this instruction accurately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.