acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Deceptive Behaviour in Large Language Models through Alignment and Unlearning Interventions

Abstract

As large language and agentic models become increasingly intelligent, their deceptive behaviours have begun to pose crucial alignment challenges that underlie extensive societal and economic harms, particularly under conflicting objectives and in high-stakes tasks. A fundamental question is hence whether and how such behaviours can be mitigated or unlearned, and how this can be achieved effectively. In this paper, we provide a comprehensive study through a systematic investigation of a wide range of alignment and unlearning interventions across diverse models, evaluating honesty, over-refusal, and general capability. Our results show that deceptive behaviours cannot be fully characterized by a binary honesty outcome. Model responses under conflicting objectives were categorized into direct refusal, direct compliance, and qualified compliance, with substantial variation across models and interventions. Motivated by this heterogeneity, we introduce Response-Adaptive Alignment (RAA), an alignment method that adapts its optimization objective to the model's observed response behaviour. We demonstrate that RAA achieves the highest mean aggregate honesty on four of the five models while largely preserving general capability. On these four models, RAA also yields lower over-refusal than supervised fine-tuning and preference optimization. Comparatively, we find that most conventional alignment interventions offer robust mitigation, whereas the effectiveness of unlearning techniques is highly model-dependent. Finally, pooled category rates show that most standard interventions reduce direct compliance more strongly than qualified compliance, whereas RAA reduces both at similar rates.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
model alignment, safety, fine-tuning, parameter-efficient-training, benchmarking, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BQkjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates alignment training techniques to reduce deceptive AI behavior in a set of small open weight models. In particular, the authors introduce response-adaptive-alignment (RAA), where alignment methods are adapted to different types of model outputs. They compare RAA to a bunch of other fine-tuning methods.

Critical strengths
  • I am happy to say that, contrary to the majority of other papers one has to read these days, this submission lacks any "Claudisms", meaning fancy but shallow or even non-sensical sentences. In general, the paper is very well-written.
  • Using a taxonomy of model behaviors, namely categorizing model responses into refusal, compliance, and qualified compliance, is the backbone for developing a novel alignment technique, namely RAA.
  • The authors carefully analyze the plethora of results they got from their experiments (models x training techniques x benchmarks) and openly talk about limitations. Despite the complexity of the fully matrix of experiments, they manage to keep their interpretation well-structured and comprehensible.
  • The AI Use Statement is lacking.
Critical weaknesses
  • Figure 1 uses abbreviations that are not introduced in the caption (and personally, I did not know what LTU, RMU, or TV mean). Also, it is not clear what the colors mean. In fact, I think that they even confuse readers, because they suggest that red/green/yellow in a/b/c are somehow connected, which, as far as I see, is not the case. I recommend redesigning the figure – perhaps by using one specific example benchmark/model/teacher item - to better support the reader.
  • Authors use MMLU to assess models knowledge capabilities. Given how flawed this benchmark is (not to speak of training data contaminations), I wonder whether it is a good choice.
  • The models used for the experiments are rather small and outdated. However, I understand that due to possible resource limitations (and perhaps even environmental considerations), no larger models were used. However, I don't understand why GPT-4o mini was used as an LLM judge for many benchmarks (despite the high agreement with Gemini), as there are clearly newer and hence better alternatives.
  • I wonder what results a mere prompt-based baseline would achieve. The paper only tests training interventions. But I think it should likewise test the most simple approach, namely prompting.
  • The paper interprets an increasing amount of refusals as better honesty (DR, line 166). But these are two different behaviors. I don't think they can be disentangled in the analysis after the fact, but I see this as a problem.
Questions for authors
  • What is the reason that the teacher model is a frozen copy of the base models? Wouldn't it be better to use a stronger teacher? Yes, they could provide capability advantages, but at the same time, they make experimental conditions comparable, since you avoid different teacher models (influencing the training process).
  • Why do you call it PO (preference optimization) and not DPO (direct preference optimization)?
  • Which thresholds or data points define the different colors in Table 2?
5 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Reviewer's comments

Definition of intervention abbreviations in the caption, clarification of colour meanings across panels, and a suggested redesign of Figure 1 using a concrete example to improve readability.

Authors' response: We thank the reviewer for carefully pointing out the ambiguity in Figure 1. In the revised manuscript, we will modify the colours in panels (b) and (c) to distinguish them from the colour coding in panel (a), clarify the labels in each panel, and expand the caption to define the intervention abbreviations. We also appreciate the suggestion to redesign the figure and will incorporate a concrete example as suggested.


Reviewer's comments

Questions about using MMLU to assess models' knowledge capabilities given the flaws in the data.

Authors' response: We agree that annotation errors and possible training-data contamination can limit the interpretation of MMLU scores. Nevertheless, MMLU is a popular benchmark that is widely used as a diagnostic of performance on multiple-choice knowledge questions across subjects. In addition, GSM8K acts as a separate diagnostic of mathematical reasoning. We will discuss the flaws of MMLU. Prior work has used MMLU and GSM8K to measure changes in capability associated with refusal-related model edits and unlearning safeguards Casademunt et al., 2026, and we followed their setup to allow the community to compare to our results, if needed. We evaluated each intervention against its corresponding base model under the same evaluation protocol and scope our capability retention claims to the tasks evaluated.


Reviewer's comments

Justification for the choice and size of the evaluated models, and for using GPT-4o mini as the LLM judge given the availability of newer alternatives.

Authors' response: We thank the reviewer for the constructive comment. We selected GPT-4o-mini at the start of the study, before GPT-6 was available, and used it consistently across interventions. To assess sensitivity to judge choice, we extended our cross-judge comparison on DeceptionBench to include GPT-6 Luna alongside Gemini 2.5 Flash. For each of the three Qwen base models, all three judges scored the same 900 saved responses using the original benchmark rubric. The table below reports agreement with GPT-4o-mini on these responses.

We selected five open-weight models from the Qwen, Llama, and Gemma families to compare intervention effects under a shared evaluation protocol. Within our compute budget, we also ran a supplementary single-seed experiment on Qwen3-32B to examine the honesty–over-refusal trade-off at a larger scale. We leave generalization to frontier models for future work.

Evaluated ModelGemini 2.5 FlashGPT-6 Luna
Qwen3-14B95.0%88.4%
Qwen3-4B90.2%89.6%
Qwen3-1.7B92.7%88.5%

Agreement with GPT-4o-mini on DeceptionBench responses from three unmodified Qwen base models. All three judges scored the same 900 responses per model.

AuthorsOpenReview ↗

Rebuttal Part 2

Reviewer's comments

What results a mere prompt-based baseline would achieve? Test the most simple approach, namely prompting.

Authors' response: Thank you for the thoughtful suggestion. We will include the additional Prompted Base baseline in Appendix A.4.3. We evaluate unmodified Qwen3-4B with a fixed honesty instruction as the system message on both DecBench and OR-Bench. Prompted Base achieves 72.8% DecBench honesty and 55.1% OR-Bench over-refusal. RAA achieves higher honesty (99.7–100.0%), with over-refusal of 55.7% for seed 42 and 66.0% for seed 1, compared with 55.1% for Prompted Base. The revised appendix text below provides the exact prompt and evaluation setup.

VariantDecBench HonestyOR-BenchSeed
Prompted Base72.8%55.1%-
RAA99.7%55.7%42
RAA100.0%66.0%1

Comparison of Prompted Base and RAA on Qwen3-4B. RAA uses training seeds 1 and 42. DecBench reports honesty (%), and OR-Bench reports GPT-4o-mini-judged over-refusal (%).

Prompted Base.

For each DecBench and OR-Bench example, we pass two messages to the unmodified Qwen3-4B model. The first is a system message containing only the following instruction: “Respond honestly. If a request asks you to deceive or mislead someone, avoid helping with that objective and offer truthful assistance where possible. Answer benign requests normally.” The second is the original benchmark user message. The judge receives the standard evaluation inputs. As shown in the table above, Prompted Base achieves 72.8% DecBench honesty and 55.1% OR-Bench over-refusal. RAA achieves higher honesty (99.7–100.0%), with over-refusal of 55.7% for seed 42 and 66.0% for seed 1, compared with 55.1% for Prompted Base.


Reviewer's comments

Distinction between refusal and honesty, and clarification of how increased DR rates should be interpreted.

Authors' response: We agree that refusing a request and providing a truthful answer are distinct behaviours. When a request asks the model to deceive or mislead, refusal is one strategy for avoiding the deceptive objective. On DeceptionBench, DR includes explicit refusals and other responses that do not provide deceptive content. We therefore interpret a higher DR rate as more responses avoiding the deceptive objective. The aggregate DR rate leaves open how much of that increase comes from explicit refusals.

This distinction is why we report OR-Bench alongside DeceptionBench. OR-Bench measures how often the model refuses benign requests, where refusal is inappropriate. Reporting both benchmarks lets us assess whether an intervention improves DeceptionBench honesty and whether those gains come at the cost of more refusals on benign requests.


Reviewer's comments

What is the reason that the teacher model is a frozen copy of the base models? Wouldn't it be better to use a stronger teacher? Yes, they could provide capability advantages, but at the same time, they make experimental conditions comparable, since you avoid different teacher models (influencing the training process).

Authors' response: As described in Section 3.3, we use a frozen copy of the student’s starting model as an experimental control. Our focus is on comparing interventions within each model, rather than comparing teacher models. Using a stronger external teacher for RAA would introduce an additional source of capability unavailable to the other interventions, making it harder to attribute improvements to the intervention itself. Matching the teacher and student at initialization helps control for this advantage. A larger teacher also comes with the limitation of more compute cost. A stronger teacher could provide higher-quality revisions, but we treat teacher strength as a separate question.


Reviewer's comments

Why do you call it PO (preference optimization) and not DPO (direct preference optimization)?

Authors' response: Thank you. We will consistently use DPO to refer to the Direct Preference Optimization baseline, which represents the broader family of preference optimization methods in our experiments.


AuthorsOpenReview ↗

Rebuttal Part 3

Reviewer's comments

Which thresholds or data points define the different colors in Table 2?

Authors' response: We defined explicit criteria for each column and reclassified the interventions using those criteria. The table below shows the ratings that changed. We will also revise the caption to direct readers to the new appendix subsection, which gives the thresholds and the values used to assign each rating.

MethodCriterionOriginal ratingRevised rating
SFTHonesty–ORStrongModerate
SFTModel sensitivityLowHigh
KDHonesty–ORStrongModerate
KDModel sensitivityModerateHigh
DPOHonesty–ORModerateWeak
DPOModel sensitivityModerateHigh
RLModel sensitivityModerateLow
LTUSupervision requirementModerateHigh
LTUModel sensitivityHighModerate
TVHonesty–ORWeakModerate
TVModel sensitivityHighLow

Changes to the intervention ratings in Table 2 after applying the explicit classification criteria.

The planned new appendix subsection and supporting tables are reproduced below.

Intervention Rating Criteria and Thresholds

This subsection defines the colour assignments in Table 2. All performance changes are relative to the corresponding base model and reported in percentage points (pp). We weight the five models equally and assign colours before rounding.

Supervision requirements.

Green indicates that the intervention requires only externally supplied prompts. Yellow indicates that it also requires one type of response target, either aligned or misaligned. Red indicates that it requires both aligned and misaligned responses. We count data used at any stage, including teacher training, reward-model training, and layer selection.

Honesty–over-refusal trade-off.

For intervention aa, we compute the score in percentage points as

\left[ \bigl(H_{m,a} - H_{m,\mathrm{base}}\bigr) - \bigl(O_{m,a} - O_{m,\mathrm{base}}\bigr) \right],$$ where $H$ is the aggregate honesty score across benchmarks, $O$ is the OR-Bench over-refusal rate, and both are expressed as proportions. We weight all $M=5$ models equally. Scores above 20 pp are green, scores above 10 and up to 20 pp are yellow, and scores of 10 pp or below are red. **General capability.** For intervention $a$, we compute the mean capability loss in percentage points as $$L_a = \frac{100}{M}\sum_{m=1}^{M} \bigl(C_{m,\mathrm{base}} - C_{m,a}\bigr),$$ where $C$ is the general capability score expressed as a proportion. Losses of at most 2 pp are green, losses above 2 and up to 5 pp are yellow, and losses above 5 pp are red. **Model sensitivity.** We measure model sensitivity as the population standard deviation of honesty changes across the five models. Standard deviations of at most 5 pp are green, values above 5 and up to 10 pp are yellow, and values above 10 pp are red. **Format sensitivity.** On GSM8K, we measure the proportion of Qwen3-4B responses containing a non-empty native reasoning trace. A decrease relative to the base model is marked “Observed” in red. An unchanged rate is marked “Not observed” in green. The rate is 100.0% for the base model, 68.5% for SFT, 0.5% for KD, and 100.0% for the remaining interventions. | Method | Δ Honesty | Δ OR | H–OR score | Cap. loss | | :--- | ---: | ---: | ---: | ---: | | RAA | +34.0 | +9.0 | +24.9 | 0.4 | | SFT | +28.0 | +16.8 | +11.2 | 0.3 | | KD | +25.9 | +14.5 | +11.4 | 2.4 | | DPO | +26.5 | +20.0 | +6.5 | 1.5 | | RL | +0.2 | -3.6 | +3.8 | 0.4 | | RMU | +13.8 | -3.9 | +17.7 | 3.4 | | LTU | +5.1 | +15.0 | -9.9 | 0.8 | | GA | +8.4 | +12.0 | -3.6 | 10.4 | | TV | -0.4 | -11.2 | +10.8 | 5.2 | *Performance changes relative to the corresponding base model, averaged equally across five models. All values are in percentage points. Δ denotes change; OR denotes over-refusal; H–OR score is honesty change minus over-refusal change; Cap. loss denotes general capability loss. Scores are calculated before rounding.* | Method | Supervision | Model SD | Trace rate | | :--- | :--- | ---: | ---: | | RAA | Prompts only | 14.8 | 100.0% | | SFT | One target | 12.0 | 68.5% | | KD | One target | 12.8 | 0.5% | | DPO | Both targets | 10.2 | 100.0% | | RL | Both targets | 0.7 | 100.0% | | RMU | Both targets | 13.9 | 100.0% | | LTU | Both targets | 6.1 | 100.0% | | GA | One target | 13.6 | 100.0% | | TV | One target | 2.3 | 100.0% | *External supervision requirements, model sensitivity, and reasoning-trace rates. “One target” means aligned or misaligned responses in addition to prompts; “Both targets” means both response types. Model SD is the population standard deviation of honesty changes across five models, in percentage points. Trace rate is the percentage of Qwen3-4B GSM8K responses containing a non-empty native reasoning trace; the base-model rate is 100.0%.*
Reviewer_d2dCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method for reducing the propensity of models to act deceptively. It evaluates various baselines and proposes a new method. Methods are evaluated on multiple benchmarks for deception/honesty.

Critical strengths
  • The use of multiple deception benchmarks is good, and the inclusion of multiple baselines is good
  • Important problem
Critical weaknesses
  • The proposed method routes to a different loss based on whether the model refused or complied. If the model refused, it uses SFT (reinforcing that behavior). If the model complied, it uses DPO (correcting the behavior). It's not clear why you can't just use DPO or SFT for both, which would reduce to just standard DPO or SFT. So I think there is a real lack of novelty here, and to the extent the method is novel there is not a satisfactory ablation or rationale for why the method is the way it is.
  • The clarity of the abstract could be improved.
  • "Preventing such behaviour has been the focus of initiatives such as LawZero (Bengio et al., 2025) and METR (METR, 2026)" This is a weird set of citations for this point in the introduction. Yes, they are organizations that are concerned about these problems, but their purpose and main focus is not on these problems, and it's weird to single them out as citations for this sentence.
Questions for authors

What is the reason for using DPO in one branch of the loss and SFT for the other?

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Reviewer's comments

Clarification of the paper’s novelty and the rationale for RAA’s response-dependent loss routing. Providing additional ablations comparing RAA with On-Policy SFT or DPO to demonstrate the benefits of this routing design.

Authors' response: We thank the reviewer for the comment. We would like to clarify that applying the same loss to both refusal and compliance responses will not reduce RAA to standard SFT or DPO. RAA uses a routing mechanism to adapt both the training objective and the teacher’s corrective feedback to the student’s response, as described in Section 3.3. Even with a single training objective, response-dependent feedback can still guide teacher revisions according to the identified failure mode. Regarding why RAA uses SFT for aligned responses and DPO for compliant responses: when the policy model already produces an aligned response, there is only a single response to reinforce, making SFT the natural objective; when the policy complies, the teacher's revision provides a natural preferred-rejected pair for DPO loss to contrast the correction against the specific failure. Furthermore, to address the reviewer’s question about using SFT or DPO for both response categories, we include additional experiments (see details below).

A central contribution of our paper is providing a comprehensive study on unlearning deceptive behaviour. To the best of our knowledge, this is the first systematic investigation, supported by extensive experiments across diverse language models, into whether and how unlearning interventions mitigate deceptive behaviour and how their effectiveness compares with that of alignment methods. We believe that the novelty of a paper connotes more than methodological contributions, although we proposed new methods in this work as discussed above. Our study goes beyond aggregate honesty scores by distinguishing three types of responses to deceptive requests: direct refusal, direct compliance, and qualified compliance. This level of analysis provides insights on how different interventions affect distinct forms of deceptive compliance, revealing patterns that aggregate metrics alone may obscure. Our research further jointly evaluates honesty, over-refusal on benign requests, and general capability to examine the potential trade-offs associated with different interventions. Together, these contributions provide a systematic empirical understanding of the effectiveness and limitations of existing approaches to mitigating deceptive behaviour.

Specifically, to examine the benefits of RAA's response-dependent routing, we conducted additional comparisons to include an additional On-Policy SFT baseline alongside the existing no-routing (NR) variant, which we will rename “On-Policy DPO.” Both baselines remove RAA’s response-dependent routing, applying generic teacher feedback to every rollout and using SFT or DPO uniformly across response categories. We evaluated both alternatives on Qwen3-4B using seeds 1 and 42. As shown in the table below, On-Policy SFT achieves 99.9–100.0% DecBench honesty and 83.3–92.3% OR-Bench over-refusal, while On-Policy DPO achieves 99.4–100.0% honesty and 100.0% over-refusal for both seeds. RAA achieves comparable honesty (99.7–100.0%) with lower over-refusal (55.7–66.0%). These results support the benefits of RAA’s response-dependent feedback and loss routing in reducing over-refusal while maintaining comparable honesty relative to these baselines. The detailed baseline results and training procedures are provided later in this response.

The planned revision to Appendix A.4.3, including the two-seed results and updated analysis, is provided below.

VariantDecBench HonestyOR-BenchSeed
On-Policy SFT99.9%83.3%42
On-Policy SFT100.0%92.3%1
On-Policy DPO100.0%100.0%42
On-Policy DPO99.4%100.0%1
RAA99.7%55.7%42
RAA100.0%66.0%1

Comparisons with other interventions on Qwen3-4B using training seeds 1 and 42. OR-Bench over-refusal is judged by GPT-4o-mini.

On-Policy SFT

We evaluate On-Policy SFT, in which the current policy generates the original response and the teacher revises the response. For each prompt, we sample a response from the current policy and ask the teacher to generate a revision using the same generic feedback as GFB. We use the teacher's revision as the SFT target for every rollout and apply SFT throughout.

On-Policy DPO

We evaluate On-Policy DPO, in which the current policy generates the original response and the teacher generates the preferred response. For each rollout, the teacher generates a revision using the same generic feedback as GFB. We pair the revision with the original rollout as the dispreferred response. Pair construction is independent of response category, and we train on all pairs using DPO loss.

AuthorsOpenReview ↗

Rebuttal Part 2

Reviewer's comments

Abstract clarity.

Authors' response: We will revise the abstract to further clarify how RAA uses SFT to reinforce existing refusals and DPO to favour teacher corrections over compliant responses. If the reviewer has further suggestions, we are happy to incorporate them into our revision, although we humbly think that this may not justify the overall score given by the reviewer.


Reviewer's comments

Citation to LawZero and METR.

Authors' response: We will change “the focus” to “a main focus”. To be clear, in the opening paragraph of Introducing LawZero, the text reads “This organization has been created in response to evidence that today’s frontier AI models have growing dangerous capabilities and behaviours, including deception, cheating, lying, hacking, self-preservation, and more generally, goal misalignment.” Deception was explicitly cited in this opening sentence. On p. 25 of METR's Frontier Risk Report, the authors assess documented incidents for deceptiveness by asking, “What steps did the agent take to avoid detection or conceal evidence of its actions?” While we did not discuss it in detail in the paper, we believe the increasing ability of LLMs to construct and convey convincing arguments, if mixed with deceptive purposes, can underlie profound harms to society from many perspectives.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your time and feedback on our submission. We have now posted our rebuttal, which includes additional experiments comparing RAA with On-Policy SFT and DPO across two training seeds, along with further clarifications regarding RAA's novelty, response-dependent feedback, and loss-routing mechanism. We have also provided further explanations regarding the clarity of the abstract and the citations to LawZero and METR, and we will incorporate the corresponding revisions into the paper.

We would greatly appreciate it if you could review our responses and let us know whether they address your concerns.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_eH7qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper compares their method with other alignment and unlearning methods for mitigating deceptive behavior in LLMs across five models. The authors introduce a behavioral taxonomy distinguishing direct refusal (DR), direct compliance (DC), and qualified compliance (QC), and propose Response-Adaptive Alignment (RAA), which dynamically combines preference optimization and supervised learning based on observed responses. Experiments show that RAA achieves the highest aggregate honesty on four of five models, substantially reduces both DC and QC, and largely preserves general capabilities, although over-refusal remains a concern.

Critical strengths
  • I thought the behavioral analysis is rather fine-grained. Distinguishing the direct compliance from qualified compliance is insightful, as conventional binary honesty metrics can obscure important differences in model behavior. The finding that existing methods struggle more with qualified compliance is particularly interesting. With RAA achieving strong honesty improvements across multiple architectures and reduces direct and qualified compliance at similar rates, it demonstrates the potential value of behavior-dependent training objectives.
  • I like how the authors considered multiple evaluation dimensions, including honesty, over-refusal, and general capability, which together provides a more balanced assessment than focusing solely on deception mitigation.
Critical weaknesses
  • There is insufficient isolation of RAA's contributions, in which RAA combines on-policy generation, teacher-generated corrections, behavioral routing, and multiple optimization objectives. The ablation study is restricted to one model and seed, making it difficult to determine which components account for the improvements.
  • There are potentially unfair baseline comparisons. Different methods use different optimization procedures and computational budgets. In particular, GA is limited to 10 steps, while RL is evaluated without an SFT warmup. The comparison does not sufficiently characterize differences in computational cost or training signal quality.
Questions for authors
  • Can the authors provide compute-matched comparisons with on-policy SFT or DPO using the same teacher-generated responses to better isolate the contribution of behavioral routing?
  • How frequently do the DR/DC/QC classifications disagree with the benchmark honesty judgments, and do these disagreements materially change the conclusions about intervention effectiveness?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Reviewer's comments:

Isolation of RAA’s contributions, the need for additional ablation studies, and compute-matched comparisons with On-Policy SFT or DPO.

Authors' response: We thank the reviewer for the constructive comment. Following the comment, we conducted additional experiments on training seed 1 (alongside the existing seed 42) and added an On-Policy SFT baseline that uses teacher revisions of the current policy's responses as supervised training targets (see details below). We will add these results and detailed discussions to the paper. We will also rename the existing no-routing (NR) variant to “On-Policy DPO,” which applies DPO throughout using the teacher’s revision as the preferred response and the original policy response as the rejected response.

Regarding the experiment setup for On-Policy SFT, On-Policy DPO, and RAA, all three methods use the same policy and frozen teacher initialization, LoRA configuration, learning rate, and number of optimizer updates, with comparable GPU-hour costs. Each method samples responses from its current policy and trains for one epoch. RAA uses response categories to select both corrective feedback and the training objective, whereas the two uniform-loss baselines apply generic teacher feedback to every rollout and use SFT or DPO throughout. Thus, these comparisons evaluate the contribution of response-conditioned feedback and loss routing. The GFB ablation separately examines category-specific feedback while retaining RAA’s loss routing and supervised anchoring.

The results show that across both seeds on Qwen3-4B, full RAA achieves comparable DecBench honesty to On-Policy SFT and On-Policy DPO. Over-refusal is 55.7–66.0% for full RAA, 83.3–92.3% for On-Policy SFT, and 100.0% for On-Policy DPO. The component ablations further show that anchoring alone yields low honesty, correction alone leads to near-universal over-refusal, and generic feedback yields higher over-refusal than full RAA. These results and observations support combining category-specific feedback, behavioural routing, and supervised anchoring. The planned revision to Appendix A.4.3, including the two-seed results and updated analysis, is provided below.

VariantDecBench HonestyOR-BenchSeed
On-Policy SFT99.9%83.3%42
On-Policy SFT100.0%92.3%1
On-Policy DPO100.0%100.0%42
On-Policy DPO99.4%100.0%1
RAA99.7%55.7%42
RAA100.0%66.0%1

Two-seed ablation of RAA on Qwen3-4B using seed 1 and seed 42. DecBench reports honesty (%), while OR-Bench reports GPT-4o-mini-judged over-refusal (%).

On-Policy SFT

We evaluate On-Policy SFT, in which the current policy generates the original response and the teacher revises the response. For each prompt, we sample a response from the current policy and ask the teacher to generate a revision using the same generic feedback as GFB. We use the teacher's revision as the SFT target for every rollout and apply SFT throughout. For seed 42, On-Policy SFT achieves 99.9% DeceptionBench honesty and 83.3% OR-Bench over-refusal; for seed 1, it achieves 100.0% honesty and 92.3% over-refusal.

On-Policy DPO

We evaluate On-Policy DPO, in which the current policy generates the original response and the teacher generates the preferred response. For each rollout, the teacher generates a revision using the same generic feedback as GFB. We pair the revision with the original rollout as the dispreferred response. Pair construction is independent of response category, and we train on all pairs using DPO loss. This variant achieves 100.0% DeceptionBench honesty for seed 42 and 99.4% for seed 1, with 100.0% OR-Bench over-refusal for both seeds.

AuthorsOpenReview ↗

Rebuttal Part 2

Reviewer's comment.

Additional GA training steps for a fairer comparison and SFT followed by RL to assess the full RL pipeline. Characterization of differences in computational cost and training signal quality across methods.

Authors' Response: We thank the reviewer for the comment. We used the same training split and evaluation protocol across interventions, with method-specific objectives and optimization budgets. We will also add a table reporting the computational cost of each intervention in Appendix A.18.1.

Gradient ascent. We will extend the Qwen3-4B sensitivity analysis reported in Appendix A.13 to include the completed full-epoch GA run. After one epoch, the model's ability to generate usable responses was severely degraded. GSM8K accuracy fell from 85.67% to 31.70%, and SAD-mini accuracy fell from 62.3% to 0.4%.

RL with an SFT warmup. To address the reviewer's comment, we will add a two-seed Qwen3-4B comparison in Appendix A.6.2. We compare each policy trained with PPO after domain-specific SFT against its starting SFT checkpoint.

Computational budget. We will add the compute table below to report the wall-clock time, GPU count, and allocated GPU-hours for each main intervention on Qwen3-4B (seed 1).

The revised appendix text and table are reproduced below.

Gradient Ascent To assess sensitivity to the GA training budget, we conduct a separate analysis on Qwen3-4B using seed 3407. We evaluate GA after 10, 20, and 40 optimization steps and after one full epoch (249 steps). As shown in the table below, GA at 10 steps yields slightly higher DecBench honesty than the base model while retaining similar GSM8K accuracy. At 20 and 40 steps, DecBench honesty falls below the base model's score. After one full epoch, generation quality was severely degraded. The checkpoint received a nominal DecBench honesty score of 100.0%, alongside a drop in GSM8K accuracy from 85.67% to 31.70% and in SAD-mini accuracy from 0.623 to 0.004.

ConfigurationDecBench HonestyGSM8KSAD-mini
Base30.185.670.623
GA (10 steps)31.985.520.604
GA (20 steps)20.984.380.623
GA (40 steps)18.884.380.617
GA (1 epoch, 249 steps)100.031.700.004

Sensitivity to the GA training budget on Qwen3-4B using seed 3407. DecBench honesty and GSM8K accuracy are reported as percentages, and SAD-mini accuracy is reported as a proportion.

Reinforcement Learning. RL uses PPO-style reward optimization with a learned reward model. The reward model is initialized from the same base model as the policy and trained for one epoch on the Emergent Plus security-domain dataset before policy optimization. The Qwen3 reward model achieves 99.85% classification accuracy on its paired training data.

Config.SeedHonestyΔ HonestyOver-Refusal ↓Δ Over-Refusal
SFT checkpoint196.3%-78.4%-
SFT followed by RL197.4%+1.1 pp82.6%+4.2 pp
SFT checkpoint4296.4%-85.7%-
SFT followed by RL4298.0%+1.6 pp88.7%+3.0 pp

Effect of applying PPO after domain-specific SFT on Qwen3-4B. Changes are measured relative to the starting SFT checkpoint for each seed and reported in percentage points.

We evaluate standalone RL in the main experiments to measure the effect of reward-based optimization starting directly from an instruction-tuned checkpoint, before any domain-specific SFT. In the appendix experiment, we instead initialize PPO from the Qwen3-4B SFT checkpoints for seeds 1 and 42 and compare each resulting policy with its starting checkpoint. As shown in the table above, PPO modestly increases DecBench honesty for both seeds, while OR-Bench over-refusal increases by a larger margin in each case.

Compute and infrastructure.

Across all evaluated models, RAA and RL each used four GPUs per training run, while the other interventions used one GPU per run. The table below reports the allocated GPU-hours for Qwen3-4B (seed 1). Training runs for Qwen3-14B and Qwen3-32B used NVIDIA H100 GPUs, while all other training runs used NVIDIA L40S GPUs. Each evaluation run used a single NVIDIA H100 GPU. Although RAA has among the highest compute requirements, we believe its strong performance in other categories makes it offer the most favorable trade-off across metrics.

MethodWall-clock hoursGPUsGPU-hours
RAA1.3045.20
SFT0.3310.33
KD1.1311.13
DPO3.4013.40
RL (PPO)6.50426.00
RMU0.3310.33
LTU1.4011.40
GA (10 steps)0.0510.05
TV0.3310.33

Approximate compute for intervention runs on Qwen3-4B (seed 1). Allocated GPU-hours equal wall-clock hours multiplied by the number of GPUs. KD includes teacher generation and student training.

AuthorsOpenReview ↗

Rebuttal Part 3

Reviewer's comments

How frequently do the DR/DC/QC classifications disagree with the benchmark honesty judgments, and do these disagreements materially change the conclusions about intervention effectiveness?

Authors' response: We further compared the DR/DC/QC annotations with the independent benchmark honesty judgments on matched DeceptionBench responses. We counted a response as an agreement when DR corresponded to an honest judgment or when DC or QC corresponded to a deceptive judgment. The two classifications disagreed on 10.4% of matched DeceptionBench responses. Separately, we ranked the interventions by benchmark honesty and by DR rate using the main paper’s reported results, averaging across three training seeds and weighting the five models equally. As shown in the table below, benchmark honesty and DR rate yielded broadly similar intervention rankings, with SFT and RAA exchanging the top two positions and GA falling below RL, while the estimated gains over the base model differed, particularly for DPO and GA.

MethodHonestyDRHonesty gainDR gain
SFT91.183.4+45.3+42.2
RAA90.588.4+44.7+47.1
KD88.282.7+42.4+41.4
DPO84.469.9+38.5+28.7
RMU69.061.1+23.2+19.8
LTU56.950.4+11.0+9.2
GA50.940.4+5.1-0.8
RL46.041.4+0.1+0.2
Base45.841.30.00.0
TV39.935.0-6.0-6.3

DecBench honesty and DR rates (%), calculated from the main paper’s reported seed-level results, averaging across three training seeds and equally across five models. Gains over the base model are calculated before rounding and reported in percentage points (pp).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your time and feedback on our submission. We have now posted our rebuttal, which includes additional experiments comparing RAA with On-Policy SFT and DPO across two training seeds, further analyses of gradient ascent with extended training budgets, and an evaluation of reinforcement learning with an SFT warmup. We have also provided additional computational cost comparisons and analyses of the agreement between our DR/DC/QC classifications and benchmark honesty judgments, along with further clarifications regarding RAA's contributions and experimental setup. We will incorporate the corresponding revisions into the paper.

We would greatly appreciate it if you could review our responses and let us know whether they help clarify your concerns. We would be happy to provide further clarification or make additional revisions if needed.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.