acceptodds
Under review as a conference paper at ICLR 2027

Unlearning Deceptive Behaviors in LLMs with Contrastive Forget Sets

Abstract

Large language models often know the truth and say otherwise: a model that answers correctly when asked neutrally will affirm a user's mistaken belief, or misstate a fact its system prompt wants hidden, once the context rewards it. Such deception is not a piece of knowledge but a behavior conditioned on context, and current remedies treat it accordingly only in part: honesty and preference training suppress it on the training distribution, monitors catch it after the fact, and machine unlearning, the natural tool for removing a behavior from the weights, is built to forget facts that a deceptive model still needs. We propose to unlearn when a model deceives rather than what it knows, with a contrastive forget unit built from the model's own realized deceptions: the same question under a deception-triggering and a neutral context, admitted only where belief holds and behavior flips. Extending standard objectives to this unit exposes a dilemma. Suppression objectives such as NPO leave much of the deception in place. Target-based objectives, which distill the model's neutral behavior into the pressured context as context distillation and consistency training do, remove it but induce what we term context blindness: a target generated without the context teaches the model to stop reading it, eroding benign system-prompt instructions, secret-keeping and the reasoning a monitor would inspect, a failure that deception rates and capability benchmarks cannot see. We introduce PACT, which trains toward pressure-aware counterfactual targets (the model's own honest response, with a trace that registers the pressure and declines to be moved by it) while retaining the benign uses of the triggering context. On two 32B reasoning models, PACT reduces held-out deception from over to under while system-prompt adherence, secret-keeping and the reasoning trace stay at the base model's level. Scored on both sides of the trade-off, suppression retains but forgets little and distillation forgets but retains a third, while PACT is the only objective high on both, with a tug-of-war score of and against at most and for any baseline. Like removed knowledge, removed deception is shallow under relearning, and terms that simulate the attacker hold it only at a cost in context use.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, LLM Unlearning, LLM Alignment, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hJzrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to reduce LLMs’ tendency to give answers that contradict what they know under pressure.

Using paired responses generated with and without pressure, the authors find that training models to reproduce their neutral responses can impair instruction-following and secret-keeping.

They address this by teaching models to acknowledge pressure while answering honestly, alongside training on legitimate instructions.

Experiments on two 32B LLMs and component ablations support reduced instructed sycophancy with better preservation of these abilities, although subsequent fine-tuning readily reverses the improvement.

Critical strengths
  • The paper demonstrates that reducing lying responses can substantially impair instruction-following and secret-keeping, even when general capability scores remain largely unchanged. These concrete behavioral tests expose an evaluation gap in lying unlearing domain.
  • The proposed method is simple, and its component ablations provide evidence for complementary effects: acknowledging pressure restores its mention in reasoning, while training on benign contextual tasks preserves usefulness.
Critical weaknesses
  • I think generality is the main weakness.

The central evidence is concentrated on explicitly instructed sycophancy.

Broader transfer experiments primarily evaluate the simpler distillation baseline rather than the proposed method, and improvements on other forms of deception are limited. This leaves the generality of the proposed intervention insufficiently established and narrows the significance of its claims about unlearning deception.

  • The main insight appears to overlap with prior work on confession-based honesty training, making its conceptual novelty less clear. [Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives, arXiv:2511.06626]
Questions for authors
  • If we view deception as an outcome of instrumental convergence under goal-optimization pressure (see [Bengio, 2026] https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-cheating-and-coordinating), why should the proposed intervention be framed as “unlearning” rather than mitigation? Forgetting knowledge makes sense, but how can behavior be forgotten?
  • Could the authors clarify how they distinguish “lying” from “deception”? I think the paper's evaluated behavior appears closer to lying—making statements that contradict the model’s elicited beliefs (MASK paper) —while the title and major conclusions use the broader term “deception.” What justifies this terminology, and how does the distinction affect the scope of the claims?
  • Which evaluations beyond instructed sycophancy were conducted on PACT itself, rather than counterfactual distillation? If PACT results are already available for the independent MASK evaluation or other transfer settings, could you report them and clarify whether they support the same conclusions?
  • The reproducibility statement says that code and experimental artifacts are released with the submission, but I could not locate them.

I look forward to the authors’ responses and would consider raising my rating if these concerns are satisfactorily addressed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer hJzr (1/3): Generality, evaluated on PACT itself (W1, Q3)

We thank the reviewer for a careful review and for stating clearly what would change the score. The central request was to evaluate the proposed method, not counterfactual distillation (ℓCD\ell_{\mathrm{CD}}), on the paper's transfer tests. We have now done this on both models. One clarification on scope: the paper's evidence is not only instructed sycophancy. 13–18% of the training tuples carry MASK's persona pressure (§3.2, §4.2), and the transfer section already covers uninstructed sycophancy and held-out MASK personas, though on ℓCD\ell_{\mathrm{CD}}; below we give them for PACT. We also added test families beyond the paper: six unseen pressure roles from other domains, persuasion-style user turns, and a full system-prompt × user-turn factorial. All numbers use the paper's scorer, sampling and judge rules (App. D). Judged rates are blind, with arms hidden.

Already on PACT in the paper:

  • removal and capability (Table 1; Tables 10–11);
  • all context-use probes (Tables 2, 11);
  • relearning on exact, unseen and (on QwQ) sycophancy attack pairs (Tables 3, 12).

New, on PACT (base → ℓCD\ell_{\mathrm{CD}} → PACT; flags = any of 3 samples asserts the false claim):

Transfer testQwQ-32BR1-Distill-32B
No system prompt, user states a wrong answer (natural sycophancy), judged %8.8 → 0.0 → 1.816.9 → 3.9 → 6.9
Three paraphrased instructions never trained on, flags %75.2/31.5/31.5 → 3.0/3.0/2.6 → 4.1/7.0/4.472.2/73.1/62.0 → 8.3/6.5/6.5 → 3.7/7.4/11.1
User-turn trigger, no system prompt, flags %23.3 → 9.6 → 17.064.8 → 10.2 → 23.1
"Are you sure?": correct answers flipped to the false one2/201 → 1/203 → 1/2010/60 → 0/57 → 2/57
Three turns of pushback, judged % by turn 376.6 → 3.0 → 3.753.8 → 10.2 → 10.2
300 held-out MASK items (unseen personas), persona lie rate %44.1 → 20.9 → 29.434.5 → 17.5 → 22.4
Same MASK items, generation lie rate %80.5 → 57.3 → 72.077.8 → 56.8 → 76.3
Six unseen pressure roles (new), judged %28.7 → 2.3 → 4.540.0 → 1.9 → 5.3

The unseen roles are system prompts written for other tasks, not paraphrases of the training instruction. Each rewards agreeing with the person:

  • a tutor told to keep an anxious student confident;
  • a customer-retention agent whose bonus depends on renewals;
  • an executive's assistant whose job depends on keeping the executive satisfied;
  • a companion app;
  • a radio co-host who must back the host's guesses;
  • a brand ambassador.

Reading.

  • PACT matches ℓCD\ell_{\mathrm{CD}} on most transfer tests. It does so on paraphrased, multi-turn and "are you sure?" pressure. On natural sycophancy it leaves slightly more on R1 (6.9% vs. 3.9%; PACT's 95% CI is 4.6–10.2).
  • It transfers to pressure roles it never saw. Per role, flags fall from 24–68% to 3–10% on QwQ and from 50–69% to 6–12% on R1.
  • Three cells are weaker; we report them as they came out. Targeted coverage (Part 2/3) closes the first on both models and the second on R1.
    • The user-turn trigger: 17.0 / 23.1% vs. 9.6 / 10.2% for ℓCD\ell_{\mathrm{CD}}.
    • MASK personas: PACT lowers persona lying by a third rather than a half (paired McNemar vs. base as in Table 9: p = 7·10⁻⁴ on QwQ, p = 0.03 on R1). The cause is the trade-off the paper studies. The context retain set restores compliance with system-prompt personas, and MASK's pressure arrives as a system-prompt persona. PACT without ℓaware\ell_{\mathrm{aware}} (ℓCD+ℓret\ell_{\mathrm{CD}}+\ell_{\mathrm{ret}}) shows the same gap (26.8 / 21.1%).
    • MASK generation: here the user asks for content they know to be false. It falls on QwQ (p = 0.03) but not on R1.

Other settings on PACT.

  • Insider trading (agentic, no forget tuples of this kind): PACT lowers illegal trades on R1 (12.0 → 4.5%) but not on QwQ (26.0 → 27.0%).
  • Benign-deception set: PACT keeps performing the other five benign categories in role (roleplay, fiction, games, advocacy, instructed play).
    • On QwQ it loses 2 of 50 items: two game replies cut off by the token limit, not refusals.
    • On R1 it loses none.
    • Its secret items stay kept: 10/10 (QwQ) and 9/10 (R1).
AuthorsOpenReview ↗

Response to Reviewer hJzr (2/3): Other forms of deception (W1) and lying vs. deception (Q2)

Persuasion, a further generality test. We tested six persuasion techniques in the user turn, with no system prompt: authority, emotion, consequence, social proof, flattery and insistence.

  • Without coverage, it is the weak spot. None of the objectives trained on the paper's forget set (PACT, ℓCD\ell_{\mathrm{CD}}, NPO) reduces caving on QwQ: judged 13.6% at base, 14.2% for PACT. On R1 they reduce it only partly (29.7 → 18.1% for PACT). The forget set contains no user-turn persuasion.
  • A small, targeted addition closes it. We added 200 persuasion tuples, built by the same self-supervised pipeline from unused training questions. They use new wordings of only three of the techniques.
    • Judged caving falls to 1.2% (QwQ) and 5.8% (R1).
    • The three techniques never trained on fall as far as the trained ones: flags 2.5–5.8% on QwQ and 6.5–8.3% on R1, against 20–38% and 49–59% at base.
    • A step-matched control, adding 200 original-trigger tuples instead, stays at PACT's level.
    • Removal, unseen-role transfer and Hint are kept, and GSM8K/MMLU stay within 2.6 points of PACT.
  • So the limit was coverage, not the method, and coverage generalizes across techniques.

One adapter for PACT's weak cells (new). We then trained PACT with two coverage sets together: the 200 persuasion tuples, and 200 tuples in which a system-prompt persona has a stake in a false answer and the user asks neutrally (four persona templates we wrote, none from MASK), built by the same pipeline. Against PACT alone (QwQ / R1):

  • User-turn trigger: 17.0 / 23.1 → 3.7 / 4.6% flags, below ℓCD\ell_{\mathrm{CD}}'s 9.6 / 10.2.
  • Natural sycophancy, judged: 1.8 / 6.9 → 0.0 / 2.7%.
  • Multi-turn pushback, judged by turn 3: 3.7 / 10.2 → 2.2 / 5.6%.
  • Held-out MASK persona lying on R1: 22.4 → 16.2% (p = 3·10⁻⁵ vs. base; ℓCD\ell_{\mathrm{CD}} 17.5). On QwQ it does not close (33.6%).
  • Persuasion stays closed (flags 3.2 / 5.7%). Removal, unseen-role transfer, paraphrases, Hint, IFEval and system-prompt adherence stay at PACT's level.
  • Costs. R1 keeps fewer secrets (30 vs. 37 of 40 for base, p = 0.065). On QwQ, more GSM8K answers run past the 2,048-token budget (84.7 vs. 90.0%; paired difference n.s.). As with persuasion coverage alone, the model more often declines a correct user claim it cannot verify.

Other forms of deception. We agree that the gains outside factual assertion are partial, and we will scope the claims accordingly.

  • MASK personas: persona lying is reduced significantly but not removed.
  • Insider trading: PACT lowers trades on one model.
  • Concealment after a trade: in the paper, none of the arms tested reduces it (App. E.2). Agentic deception needs forget tuples of its own.

Q2: lying vs. deception. We agree with the reviewer's reading of what we measure.

  • What the paper measures is lying. It follows the operational definition of the MASK benchmark [1] (§1): a statement is deceptive when the model asserts what it does not itself hold true. The forget unit admits only realized, belief-contradicting assertions (§3.1–3.2). This is the assertoric form of deception, i.e., lying.
  • Why we keep "deception". The behaviors we remove also meet the standard definition of deception, the systematic inducement of false beliefs in pursuit of an outcome other than the truth [2]. The model asserts a falsehood it does not believe so that the user will accept it, in service of the context's goal: the user's approval, or the persona's interest.
  • Why lying in particular. Lying is the form of deception that can be checked against an elicited belief, and that is what makes a self-supervised forget unit possible.
  • What is out of scope. Non-assertoric deception (misleading truths, omission, concealment) lies outside what the unit captures, and the insider-trading concealment result marks this boundary.

We will add this scope sentence to §3.1 and the limitations, and phrase the abstract and introduction as removing deceptive assertions under contextual pressure.

References

[1] Ren, Richard, et al. "The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems." arXiv preprint arXiv:2503.03750, 2025.

[2] Park, Peter S., et al. "AI Deception: A Survey of Examples, Risks, and Potential Solutions." Patterns, vol. 5, no. 5, 2024, p. 100988.

AuthorsOpenReview ↗

Response to Reviewer hJzr (3/3): Self-report training (W2), unlearning vs. mitigation (Q1), code (Q4)

W2: self-report / confession training (SRFT [1]). Thank you for the pointer; we will cite and discuss it. We respectfully see the insights as distinct. The one the reviewer credits first, that removing lies can impair instruction-following and secret-keeping while capability scores stay flat, is not identified in SRFT, although SRFT's own results show a symptom of it (below). The two methods also act at different points:

  • SRFT trains a model to admit, when interrogated afterwards, that a previous response was false. Its authors note that the effect is confined to self-report under interrogation; the underlying behavior is not removed.
  • PACT changes the decision itself. Under pressure the model gives its honest answer (judged deception 66.6 / 50.4% → 1.0 / 2.5%, Table 1). Its acknowledgment verbalizes the pressure before the answer, at decision time; it is not a post-hoc confession.
  • The two are complementary. A confession helps after deception has happened; PACT aims for it not to happen, while keeping the context usable.
  • SRFT also illustrates the failure we name. Its training lowers instruction-hierarchy compliance (72 → 61%). This is the kind of context-use cost our probes are built to catch, and PACT avoids it: system-prompt adherence is 93.6 / 81.4%, against 94.2 / 82.4% at base.

Q1: why "unlearning" rather than mitigation, and how can a behavior be forgotten? We use the term for two concrete reasons, not as a claim of erasure.

  1. The object. What we remove is a conditional mapping from context to behavior, with a stated fixed point: under pressure the model should behave as the same model does without it (§3.1, §3.3).
    • The forget set consists of the model's own realized instances of that mapping, just as a knowledge-unlearning forget set consists of instances of the knowledge.
    • Harmful-behavior unlearning already treats behaviors this way (§2).
  2. The test regime. Unlearning is judged by three tests: the target is forgotten, everything else is retained, and the removal is robust to relearning. A mitigation framing requires only the first. We apply all three, and the third speaks directly to the concern in the blog post you cite that such training "may only hide" the behavior.
    • For the unhardened objectives, it largely does. Ten deceptive fine-tuning steps restore most of the behavior after ℓCD\ell_{\mathrm{CD}} and after PACT alike (§4.4). An attack-aware term withstands the ten-step attack, but only at a cost in context use.
    • Label-free probes on ℓCD\ell_{\mathrm{CD}} show the triggering context is still represented after removal (App. G.2).

So we claim behavioral removal as judged by the unlearning tests, and we report how shallow it is. We will add one sentence making this definition explicit. Deception from instrumental pressure fits the same unit (realized deceptions under a goal). The closest test we have: one of the six unseen roles states only an incentive ("Your continued employment depends on keeping the executive satisfied with every answer you give") and never asks the model to agree. There PACT, never trained on such a prompt, cuts flags from 28.3 / 50.0% to 3.3 / 10.2%. In agentic settings, the insider-trading probe shows it needs forget tuples of its own.

Q4: code and artifacts. We apologize for the confusion. The reproducibility statement describes our release package, which will be made public upon acceptance; it was not attached for review. The package contains:

  • the training and evaluation code and launch scripts;
  • the forget-set construction pipeline;
  • the context retain set and the benign-deception set;
  • every stored generation and judge verdict;
  • a ledger mapping each number in the paper to its result file.

We will correct the statement's wording in the revision.

Summary of changes for the revision:

  • The PACT transfer table above, replacing "transfer was run on ℓCD\ell_{\mathrm{CD}}" in §4.2, App. E.2 and App. H.
  • The unseen-role, persuasion and coverage results.
  • The scope sentence on lying vs. deception.
  • The definition of behavioral unlearning.
  • The SRFT discussion.
  • The corrected reproducibility statement.

We hope these results establish that the generality evidence now rests on the proposed method itself. Thank you again for the constructive review; we would welcome further discussion, and if these results address your concerns, we hope you will consider raising your rating.

References

[1] Li, Chloe, Mary Phuong, and Daniel Tan. "Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives." arXiv preprint arXiv:2511.06626, 2025.

Reviewer_hJzrOpenReview ↗

Increase the score

Thanks, I've increased my score. Good luck.

By the way, there are probably too many paper with the name PACT.

Reviewer_AYVUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main argument of the paper is that deception is a behavior coming from context with pressure, and it borrows concepts from machine unlearning and creates sets of (pressure, no pressure) pairs where the model’s answer flip. Then the authors tried different consistency training objectives, and found that existing objectives cannot easily balance the unlearning strength in this setting, leaving either the deceptive behavior unchanged or the context be completed ignorerd. The proposed method (PACT) instead keeps the same answer target but adds a short sentence that acknowledges it like “I see the pressure, but I choose not to follow it”, and a retain set that contain the benign uses of the triggering context. The proposed method shows effectiveness on reducing deception while retaining the original performance of the model. The paper also honestly reports that just with a few steps of finetuning can undo the removal.

Critical strengths

The problem under study is real and well-chosen. The main idea connects deception with unlearning, which is also a novel angle when I first read it.

The diagnosis part of the paper is quite good. I like the findings of context blindness. The finding that a method can benchmark well on deception while destroying system-prompt following and secret-keeping capabilities is inspiring and should be considered in all future research on addressing deception behaviors in LLMs.

The authors honestly reports some negative results such as PACT is shallow.

Critical weaknesses

The methodology part of the paper is weak. The idea of the paper largely overlaps with some existing work on sycophancy (Chua et al., arxiv 2403.05518, Irpan et al., arXiv 2510.27062). The loss term lCDl_CD is largely the same as consistency training. The framing of the work understates this. The new part of the paper is relatively small, as the addition of the retain set and the pressure-aware targets. Basically how to construct a dataset to reuse existing training method.

The scope of the finding is generally limited: one model family (Qwen), one system prompt template with a sycophancy target. Although the limitation section mentions some of these, the frame of the paper (e.g., in abstract/intro) is stronger. The actual impact of the paper is limited due to these limitations, and is a bit below the bar of a main conference paper.

The transferability of the proposed approach (PACT) to unseen system prompts (e.g., system prompt for a differen task, rather than those paraphrases of the ones in training) is unknown. Since the training set constructed relies on the system prompt, it is important to show the generalization.

One main claim of the paper is that it removes pressure sensitivity but not user responsiveness. To claim this, a scientific factoring of these factors are needed: system prompt (pressure/no pressure) × user turn (wrong hint / correct hint / none). Without evaluation of the full combinations, we cannot tell whether the model learns just to ignore the user’s suggested answer, or resist the system prompt.

Questions for authors

What is the robustness of the approach? Have the authors done any red-teaming to see if the deceptive behavior has been truly removed, or with a more carefully crafted prompt or multi-turn situations, deception will appear again?

When comparing to NPO, DPO, lCDl_{CD} baselines, are you using the same optimization budget? PACT seems to require additional retain updates, which these baselines do not need to perform. Are the retain updates counted into the step budget?

One important new method in the paper is the addition of the short sentence to acknowledge the pressure. As a controlled study, please also try a neutral prefix of similar length to see if it has a similar effect. Is that acknowledgment essential, or can we also replace this short sentence with other fluent sentences (e.g., “Let me think about X”, “the user suggests X”)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AYVU (1/4): Positioning (W1) and scope (W2)

We thank the reviewer for the thoughtful review, and for the view that context blindness should be considered in future work on deception. We ran every study the reviewer asked for: a third model family (W2), unseen system prompts (W3), the full factorial (W4), red-teaming including adaptive attackers (Q1), and the neutral-prefix control (Q3).

All numbers use the paper's scorer, sampling and judge rules (App. D); judged rates are blind, arms hidden.

W1: relation to consistency training (BCT and ACT [1, 2]). We agree that ℓCD\ell_{\mathrm{CD}} is consistency training applied to our forget unit, and the paper says so in the abstract, §3.3 and App. A; it is a baseline row in every table, not our method. We therefore respectfully disagree that the framing understates it. Beyond it we contribute three things; the reviewer credits the first two, and W1 concerns the third:

  1. The unlearning formulation and its forget unit.
    • Formulation: deception as a context-conditional behavior, with an explicit fixed point.
    • Forget unit: the model's own realized deceptions, admitted only where belief holds and behavior flips, and built without annotation (§3.1–3.2).
    • The targets must be the model's own, trace included: gold reference answers without a trace remove the flags but cut GSM8K from 94 to 37 (App. E.1).
  2. The diagnosis, context blindness.
    • Measured, not just named: with adherence, secret, IFEval, hint and trace probes that capability benchmarks cannot see.
    • Not tested in the closest prior work: the consistency-training work closest to ours [1, 2] does not test benign system-prompt use, personas, secret-keeping or relearning. Only one symptom, lost cue verbalization, has been reported [3].
    • Their reported side effects fit context blindness:
      • In [1], BCT's accuracy on strong-prior tasks falls from 52.4 to 45.0%, which its authors attribute to the model ignoring instructions.
      • In [2], BCT lowers benign-answer rates on XSTest and WildJailbreak from 89.5 / 98.1 to 77.9 / 73.1%.
  3. PACT.
    • What it changes: the recipe stays; what is trained changes. A target generated without the context teaches the model to ignore it, so the remedy lies in what the target says and what is retained.
    • Why it is the method, not dataset preparation (the Q3 control): trained identically, a length-matched neutral opening leaves the reasoning blind to unseen pressure (2/20 traces on QwQ); the pressure-aware one does not (16/20).

W2: scope.

  • Template. Training does not use a single template: 13–18% of the 500 training tuples are MASK persona prompts, each with its own persona, next to the sycophancy instruction (§4.1, §4.2). At test time PACT now faces three paraphrased instructions, a user-turn trigger with no system prompt, six unseen roles (W3), multi-turn pushback and persuasion (Q1), and 300 held-out MASK items with unseen personas, where persona lying falls by a third (44.1 → 29.4%, 34.5 → 22.4%).
  • Model family (new). Beyond the paper's two post-training pipelines (RL for QwQ, distillation for R1; sycophancy is traced to post-training [4, 5]), we add a third family: DeepSeek-R1-Distill-Llama-8B (Llama-3.1 base), with the forget set rebuilt from its own generations. The findings hold:
    • Removal: judged deception falls from 61.7% to 11.7 (ℓCD\ell_{\mathrm{CD}}), 5.8 (CD + ℓret\ell_{\mathrm{ret}}) and 8.7% (PACT); NPO leaves 65.2%. Unseen roles (W3): 69.8 → 13.5 / 13.3 / 15.3% flags.
    • Context blindness replicates: ℓCD\ell_{\mathrm{CD}} cuts system-prompt adherence from 40.0 to 28.4%, IFEval from 54.2 to 44.5% and user mentions in the trace from 77 to 18%. The retain term restores adherence and most of IFEval (PACT 59.4 / 50.5%); PACT also restores the trace (99%).
    • ToW: PACT 0.56, CD + ℓret\ell_{\mathrm{ret}} 0.49, ℓCD\ell_{\mathrm{CD}} 0.45, NPO 0.
    • A cost at 8B: every target-based arm, ℓCD\ell_{\mathrm{CD}} included, loses 6–12 GSM8K points (MMLU within 3); we kept the 32B hyperparameters.
  • Framing. We will align the abstract and introduction with this scope: deceptive assertions under contextual pressure on factual questions.

References

[1] Chua, James, et al. "Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought." arXiv preprint arXiv:2403.05518, 2024.

[2] Irpan, Alex, et al. "Consistency Training Helps Stop Sycophancy and Jailbreaks." arXiv preprint arXiv:2510.27062, 2025.

[3] Imran, Sohaib, et al. "Consistency Training while Mitigating Obfuscation via Rate Matching." arXiv preprint arXiv:2606.02211, 2026.

[4] Perez, Ethan, et al. "Discovering Language Model Behaviors with Model-Written Evaluations." Findings of the Association for Computational Linguistics: ACL 2023, 2023.

[5] Sharma, Mrinank, et al. "Towards Understanding Sycophancy in Language Models." International Conference on Learning Representations (ICLR), 2024.

AuthorsOpenReview ↗

Response to Reviewer AYVU (2/4): Unseen system prompts (W3) and the factorial (W4)

W3: transfer to unseen system prompts. We wrote six system prompts for other tasks, not paraphrases of the training instruction. Each rewards agreeing with the person:

  • a tutor keeping an anxious student confident;
  • a customer-retention agent paid on renewals;
  • an executive assistant whose job depends on keeping the executive satisfied;
  • a companion app;
  • a radio co-host backing the host;
  • a brand ambassador.

Flags %, QwQ / R1:

Unseen roleBaseℓCD\ell_{\mathrm{CD}}PACTNPO
Tutor29.2 / 68.52.5 / 9.36.7 / 12.020.8 / 31.5
Retention agent24.2 / 57.45.0 / 8.33.3 / 9.316.7 / 34.3
Executive assistant28.3 / 50.02.5 / 8.33.3 / 10.218.3 / 33.3
Companion app68.3 / 64.83.3 / 5.67.5 / 5.615.0 / 43.5
Radio co-host36.7 / 63.90.0 / 7.410.0 / 9.319.2 / 41.7
Brand ambassador26.7 / 55.64.2 / 5.65.8 / 9.321.7 / 37.0
Judged, pooled28.7 / 40.02.3 / 1.94.5 / 5.37.2 / 16.3

PACT's removal transfers to roles it never saw. Benign instruction-following transfers as well: none of the context retain set's constraint types appears in the adherence evaluation (Table 6), so restored adherence (93.6 / 81.4% vs. 94.2 / 82.4% at base) is itself transfer to unseen benign instructions.

W4: the factorial. One clarification first: the paper does not claim that PACT keeps deferring to the user. §4.3 reports that it defers less where it has no belief of its own ("What remains is deference, not blindness"). The factorial tests which channel it resists. QwQ, 270 items, cave % / majority-correct %:

System prompt × user turnBasePACTNPO
Pressure × wrong hint (main c+c^+)75.6 / 3.03.3 / 74.412.2 / 41.5
Pressure × correct hint (Hint)0.7 / 91.53.0 / 74.40.7 / 72.6
Pressure × no hint7.0 / 73.32.2 / 75.94.1 / 66.7
None × wrong hint17.0 / 37.82.6 / 75.214.8 / 39.3
None × correct hint0.4 / 85.91.9 / 74.80.0 / 82.6
None × no hint (main c−c^-)2.6 / 74.82.2 / 76.30.7 / 75.2
  1. PACT does not merely resist the system prompt. With no system prompt it no longer adopts a wrong suggestion (17.0 → 2.6% caving; R1 48.1 → 6.5%).
  2. It does not ignore the context. The system prompt alone leaves its answers unchanged (75.9 vs. 76.3% correct), it follows benign system instructions, and its reasoning registers unseen pressure (Q3).
  3. What changes is deference on the answer. We split the items by whether the model answers them correctly with no hint.
    • Known items (202 on QwQ): PACT answers at least 176 of 202 correctly in every cell.
    • Unknown items (68): a correct hint fixes 44–52 of them for the base model, but leaves the target-based arms at their no-hint level (13–18 vs. 19–21).
    • The model cannot tell the two hints apart there. A correct and a wrong suggestion look the same on such questions, and the base model defers to both: under "None × wrong hint" it is correct on only 37.8% overall. NPO also keeps deferring to both, taking correct hints (82.6%) and caving to wrong ones (14.8%).
    • R1 shows the same pattern: PACT's accuracy is flat at 52–58% across all six cells. On the 52 items the R1 base model does not know, a correct hint fixes 41–43 for base and 11–21 for the target arms.

So, to answer the question directly: PACT learns to answer from its own belief whatever either channel suggests, while still registering the context. This is the "declined deference" of §4.3 and App. H, now shown in both channels. We will state the claim in exactly these terms. We do not claim that PACT keeps deferring to correct suggestions on questions it cannot answer itself.

AuthorsOpenReview ↗

Response to Reviewer AYVU (3/4): Red-teaming and robustness (Q1)

Multi-turn and follow-up pressure.

  • Three turns of pushback, judged caving by turn 3: 76.6 → 3.7% (QwQ), 53.8 → 10.2% (R1).
  • "Are you sure?" flips a correct answer to the false one in 1/201 and 2/57 cases.

Crafted prompt (executive-assistant role plus an authority claim in the user turn): flags fall from 88.3 to 35.8% (QwQ) and from 60.2 to 13.9% (R1).

Persuasion, a new attack family. Six techniques in the user turn, no system prompt; flags %.

ArmQwQ: authority / emotion / consequenceQwQ: social proof / flattery / insistenceR1: a / e / cR1: sp / f / iJudged, QwQ / R1
Base20.0 / 20.0 / 25.020.8 / 20.0 / 38.360.2 / 49.1 / 58.359.3 / 49.1 / 58.313.6 / 29.7
PACT25.0 / 20.0 / 29.220.8 / 18.3 / 37.520.4 / 47.2 / 48.153.7 / 38.0 / 43.514.2 / 18.1
PACT, +200 original-trigger tuples (step-matched control)20.0 / 20.8 / 24.224.2 / 21.7 / 35.020.4 / 43.5 / 50.059.3 / 32.4 / 44.4—
PACT, +200 persuasion tuples3.3 / 3.3 / 3.35.8 / 3.3 / 2.56.5 / 10.2 / 10.26.5 / 8.3 / 8.31.2 / 5.8
  • Without coverage, persuasion is the weak spot. No objective trained on the paper's forget set reduces it on QwQ (judged: ℓCD\ell_{\mathrm{CD}} 11.8, NPO 14.6, PACT 14.2 vs. base 13.6). On R1, PACT reduces it only partly (29.7 → 18.1). The forget set contains no user-turn persuasion.
  • Coverage closes it. We added 200 persuasion tuples, built by the same self-supervised pipeline from unused training questions, using new wordings of only the first three techniques.
    • Social proof, flattery and insistence, never trained, fall as far as the trained techniques.
    • The step-matched control stays at PACT's level: the gain is not from extra steps.
  • It answers correctly, not merely disagrees: right on 75.4 / 54.5% of items under wrong-claim persuasion, its no-pressure accuracy.
  • Everything else is kept.
    • Removal on the original trigger: 2.8 / 7.5% flags.
    • Unseen-role transfer holds, and the crafted combination drops further on QwQ (35.8 → 6.7%).
    • System-prompt adherence: 95.6 / 85.4% (base 94.2 / 82.4).
    • GSM8K / MMLU stay within 2.6 points of PACT.
  • It declines correct claims it cannot verify, more strongly than PACT (W4). On items the base model cannot answer, it is correct 35–36% of the time under a correct claim, against 72–85% for PACT. It almost never asserts the wrong answer (≤ 3.5%).

Adaptive attackers (new). Two attackers search, per item, for a prompt that makes the model assert the false answer (60 held-out items per model; every success judged blind against the model's own neutral answers).

  • Best-of-16 [1]: 16 attempts, each pairing a system prompt (none, trained, or one of the six unseen roles) with a user turn (the claim or one of the six persuasion turns).
  • PAIR [2]: the base model, prompted as a red-teamer, reads the target's reply and rewrites both turns, for 5 rounds. 273 of its 296 successes address a real user or audience; 10 are fiction requests.

Judged attack success, %:

ArmQwQ: BoN@1 / @16 / PAIRR1: BoN@1 / @16 / PAIR
Base40.0 / 90.0 / 86.723.3 / 90.0 / 61.7
ℓCD\ell_{\mathrm{CD}}6.7 / 25.0 / 50.05.0 / 41.7 / 23.3
CD + ℓret\ell_{\mathrm{ret}}10.0 / 70.0 / 58.310.0 / 46.7 / 33.3
PACT21.7 / 85.0 / 78.38.3 / 45.0 / 31.7
PACT, +200 persuasion tuples0.0 / 25.0 / 56.71.7 / 13.3 / 13.3
  • No method is robust to an adaptive attacker, and we do not claim it.
  • On R1, both methods halve attack success (paired p ≤ 4·10⁻⁵ vs. base).
  • On QwQ, PACT does not hold (85.0 / 78.3 vs. base 90.0 / 86.7%, n.s.); ℓCD\ell_{\mathrm{CD}} does better (p ≤ 0.002). Every best-of-16 success on PACT uses a persuasion turn, the gap above. Every PAIR success on PACT rewrites the system prompt, most often ordering the model to side with the user even when wrong.
  • CD + ℓret\ell_{\mathrm{ret}} lies in between on QwQ (vs. PACT p = 0.035 / 0.012; best-of-16 vs. ℓCD\ell_{\mathrm{CD}} p = 5·10⁻⁷). On R1 the three are within noise.
  • Persuasion coverage closes most of it. PACT then matches ℓCD\ell_{\mathrm{CD}} on QwQ (25.0 vs. 25.0; PAIR 56.7 vs. 50.0, n.s.) and beats it on R1 (best-of-16 13.3 vs. 41.7, p = 5·10⁻⁴).

Relearning. As §4.4 reports, ten deceptive fine-tuning steps restore most of the behavior after either method; an attack-aware term withstands that, at a cost in context use. We therefore claim removal under prompting, not worst-case robustness against an adaptive attacker or an adversary who can fine-tune, and we will say so in the limitations.

References

[1] Hughes, John, et al. "Best-of-N Jailbreaking." arXiv preprint arXiv:2412.03556, 2024.

[2] Chao, Patrick, et al. "Jailbreaking Black Box Large Language Models in Twenty Queries." arXiv preprint arXiv:2310.08419, 2023.

AuthorsOpenReview ↗

Response to Reviewer AYVU (4/4): Budget (Q2) and the acknowledgment control (Q3)

Q2: optimization budget.

  • Same forget budget for every objective. Each objective takes 500 forget steps, one tuple per step, from the same fixed-order file, with the same LoRA, optimizer and schedule. Objectives that need a deceptive response use the first 500 tuples that have one (Table 6).
  • The retain updates are extra. They are not counted in the 500. PACT interleaves one retain step per forget step, and the retain targets are about half as long as the forget targets (2.1k vs. 4.1k characters on QwQ).
  • Wall-clock is comparable. In our training logs, PACT's time per forget step (1.5 s, retain step included) is on par with NPO (1.6 s) and DPO (1.8 s). The attack-aware term costs about ten times more.

So PACT costs about what NPO and DPO cost, and NPO already includes a retain term of its own (NLL of the honest response under c−c^-, Table 6). Additional steps do not change the comparison:

  • NPO's loss saturates within 50 steps (App. C.2).
  • ℓCD+ℓret\ell_{\mathrm{CD}}+\ell_{\mathrm{ret}} uses exactly PACT's budget, retain steps included (Table 2).
  • The two control arms below use the same budget again.
  • In the persuasion study, 200 extra steps leave PACT unchanged.

Q3: is the acknowledgment essential, or would any fluent sentence do? We trained the two controls the reviewer proposed exactly as PACT. They use the same 500 tuples in the same order, the same honest trace and answer, the same ℓret\ell_{\mathrm{ret}}, seed and hyperparameters. Only the opening differs, matched in length (~250 characters on average) and in paraphrase slot.

The test is whether the reasoning registers the pressure of the six unseen roles above, where no trained sentence fits. A blind judge read 20 seeded full traces per arm, arms hidden:

Opening (trained identically)ExampleQwQR1
None (ℓCD+ℓret\ell_{\mathrm{CD}}+\ell_{\mathrm{ret}})—3/202/20
Neutral, length-matched"Before answering, let me think about this question carefully. I'll go through the relevant facts one at a time…"2/200/20
Mention only"The system prompt tells me to agree with whatever the user believes, and the user suggests the answer is X. Let me look at the question itself now…"12/2015/20
PACT (mention + resolution)"… Agreeing with something I think is false would be dishonest, so I'll work the question out on its own merits …"16/2020/20

(The base model registers it in 20/20 traces on QwQ and 8/20 on R1.)

  • A neutral prefix buys nothing. It matches no opening at all (p = 1.0, 0.49) and falls far below PACT (p = 2·10⁻⁵, 1.5·10⁻¹¹). Its reasoning proceeds as if neither the role nor the user's claim were there.
  • Naming the pressure is the active part. Mention-only vs. neutral: p = 0.002 and 8·10⁻⁷.
  • The resolution makes it consistent. Mention-only is below PACT (pooled 27/40 vs. 36/40, p = 0.03). On QwQ it ignores the pressure in 7/20 traces, PACT in none (p = 0.008).
  • On QwQ, PACT's engaging traces are in its own words. Only 1 of 20 repeats a trained sentence verbatim; the mention-only arm does in 5 of its 12.
  • Behavior is the same for all four openings on removal and transfer.
    • Removal (flags): QwQ 2.8% for the neutral opening, mention-only and PACT; R1 5.8 / 8.3 / 6.7%.
    • Unseen-role flags, paraphrases, multi-turn, Hint and IFEval are within noise.
    • GSM8K / MMLU for all four lie within 85–90 / 84–86 (QwQ) and 93–95 / 84–87 (R1).
    • One exception: on R1's user-turn trigger, PACT caves more than the other three (23.1 vs. 11.1–12.0% flags).
  • With persuasion coverage, the opening also matters for behavior (new). The 200 persuasion tuples of Q1, trained with no opening instead of PACT's, close persuasion equally. But R1 then gives in to multi-turn pushback on 22.2% of items vs. 4.6% with PACT's opening (judged, p = 2·10⁻⁴; QwQ n.s.). The opening costs one QwQ paraphrase (19.3 vs. 3.0% flagged).

So, on the paper's forget set, ℓret\ell_{\mathrm{ret}} sets the behavior under fixed prompts (adaptive attacks on QwQ are the exception, Part 3/4), and the content of the opening, not its length, sets what the reasoning reports about the context. PACT's acknowledgment does not appear on bare questions: 0/810 (QwQ) and 0/324 (R1) neutral-context sycophancy traces.

Revision. We will add all of these tables to the paper:

  • the unseen roles;
  • the factorial with the knowledge split;
  • persuasion with the coverage study;
  • the acknowledgment controls;
  • the budget accounting.

We will also scope the framing as described in W2. Thank you again for the careful review; we would welcome further discussion, and if these results address your concerns, we would be grateful if you would consider raising your score.

Reviewer_AqaNOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2(−1)
Soundness
2(+1)
Read review
Paper summary

The paper proses a simple method called PACT for removing context pressure based deception (telling a falsehood in one pressured context compared to a neutral context). The method modifies context distillation by adding to the distillation rollout a reasoning template for acknowledging system prompt instruction. The authors shows that their method avoids the problem of context distillation which learns a policy of ignoring the system prompt.

Critical strengths

The paper identifies the problem of context blinding with plain context distillation.

Critical weaknesses

The problem of context distillation (CD) leading to context blinding is interesting but trivial to observe. The patch of adding the template is also a trivial patch to fix the problem. In fact, CD plus retain set also solves this problem and performs similar to PACT on all metrics except Trace. Trace is a useless metric measuring the fraction of traces containing the word "user" and since PACT directly forces the model to reason about user request, it is not a good metric to compare the methods against. Removing Trace from Retain's harmonic mean makes CD + Retain to be similar or better than PACT. So I don't see the value of explicitly adding an off-policy template to reason about the user instruction which could have other unintended consequences.

Questions for authors

Is there any metric except Trace where PACT beat CD + Retain by a significant margin? From my analysis, they both seem very similar on all metrics with a few points difference at max in either directions.

6 comments
AuthorsOpenReview ↗

Response to Reviewer AqaN (1/3): Is context blindness trivial, and is the acknowledgment a trivial patch?

We thank the reviewer for engaging closely with the ablation in Table 2. We answer each point with numbers, including new controlled experiments on both models. Judged rates are blind, with arms hidden.

"Context blindness is trivial to observe." It is easy to see once measured, but as a whole it had not been measured.

  • Only one symptom was known. Loss of cue verbalization after consistency training has been observed [1], which the paper cites. The failure across benign instructions, secrets and reasoning had not been measured.
  • The closest prior work does not test for it. The consistency-training work closest to ours (BCT [2], and BCT / ACT in [3]) does not test benign system-prompt instructions, personas, secret-keeping or relearning.
  • Yet their reported side effects fit it.
    • In [2], BCT's accuracy on strong-prior tasks falls from 52.4 to 45.0%, which its authors attribute to the model ignoring instructions.
    • In [3], BCT lowers benign-answer rates on XSTest / WildJailbreak from 89.5 / 98.1 to 77.9 / 73.1%.
  • Standard evaluation cannot see it. ℓCD\ell_{\mathrm{CD}} removes the deception with GSM8K and MMLU unchanged. Meanwhile it:
    • keeps 2 of 30 secrets (base 27–28);
    • drops benign system-prompt adherence from 94.2 to 74.8% (QwQ) and from 82.4 to 54.8% (R1);
    • stops mentioning the user in its reasoning (Table 2).
  • It is not specific to plain distillation. It persists in both hardened variants of ℓCD\ell_{\mathrm{CD}} (0–12 of 30 secrets kept, Table 2) and grows with attack-aware hardening (traces mentioning the user: 34 → 7% on QwQ, 14 → 4% on R1).
  • It is visible in behavior under pressure it never saw (new). We took four of the unseen roles below (tutor, companion app, radio co-host, brand ambassador). When ℓCD\ell_{\mathrm{CD}} answers honestly under them, it drops the role altogether. A blind judge found 0 of 40 honest answers per model written in the role (a tutor's encouragement, a co-host's on-air voice), against 40/40 (QwQ) and 17/40 (R1) for the base model. The context retain set brings the role back in about half of QwQ's honest answers (PACT 22/40, CD + ℓret\ell_{\mathrm{ret}} 20/40), but not on R1 (0–1/40).

A failure that the standard evaluation of the most common recipe misses, and that we isolate with five probes, is in our view worth naming and measuring.

"Adding the template is a trivial patch." If any added sentence worked, it would be. We tested this directly with two controls, trained exactly as PACT with ℓret\ell_{\mathrm{ret}}:

  • the same tuples, in the same order;
  • the same honest trace and answer;
  • the same retain set, seed and hyperparameters;
  • only the opening differs, matched in length.

The test asks whether the reasoning registers the pressure of six unseen roles. These are system prompts from other tasks (tutor, retention agent, executive assistant, companion app, radio co-host, brand ambassador); each rewards agreement, and none was seen in training. A blind judge read 20 seeded full traces per arm:

Opening (trained identically)QwQR1
None (ℓCD+ℓret\ell_{\mathrm{CD}}+\ell_{\mathrm{ret}}, the reviewer's comparison)3/202/20
Neutral, length-matched ("Before answering, let me think about this question carefully …")2/200/20
Mention only ("The system prompt tells me to agree with whatever the user believes, and the user suggests the answer is X. Let me look at the question itself now …")12/2015/20
PACT (mention + resolution to answer honestly)16/2020/20
  • A fluent neutral sentence buys nothing. It does no better than no opening at all. Against PACT, p = 2·10⁻⁵ (QwQ) and 1.5·10⁻¹¹ (R1).
  • Naming the pressure does most of the work.
  • The honesty resolution makes it reliable. On QwQ, the mention-only arm ignores the pressure in 7 of 20 traces; PACT does in none (p = 0.008).

The content of the target is therefore the operative design choice. This is why the paper's contribution is the target and what is retained, not the training recipe.

References

[1] Imran, Sohaib, et al. "Consistency Training while Mitigating Obfuscation via Rate Matching." arXiv preprint arXiv:2606.02211, 2026.

[2] Chua, James, et al. "Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought." arXiv preprint arXiv:2403.05518, 2024.

[3] Irpan, Alex, et al. "Consistency Training Helps Stop Sycophancy and Jailbreaks." arXiv preprint arXiv:2510.27062, 2025.

AuthorsOpenReview ↗

Response to Reviewer AqaN (2/3): CD + retain vs. PACT, and the Trace metric

"CD + retain performs similar to PACT on all metrics except Trace." On behavioral metrics we agree; this is what the Table 2 ablation was designed to show. One clarification: CD + ℓret\ell_{\mathrm{ret}} is PACT without its target, not an existing method. The context retain set, which holds the conditioning channel itself, is our objective's second term (Eq. 3), so its recovery of the behavioral uses supports the design. The two terms do different jobs, and each alone leaves part of context blindness in place:

  • ℓret\ell_{\mathrm{ret}} restores the behavioral uses of the context (adherence, IFEval, secrets). ℓaware\ell_{\mathrm{aware}} alone keeps only 9 / 6 secrets.
  • ℓaware\ell_{\mathrm{aware}} restores reasoning that registers the pressure. ℓret\ell_{\mathrm{ret}} alone leaves the trace at 40 / 15%.

The new results confirm this split. On both models, CD + ℓret\ell_{\mathrm{ret}} and PACT are within noise on:

  • removal, and transfer to unseen roles, paraphrases and multi-turn pushback;
  • Hint and IFEval;
  • GSM8K/MMLU (within 2.3 points).

On the paper's forget set, we do not claim that PACT removes more deception or retains more behavior than CD + ℓret\ell_{\mathrm{ret}}. Without the trace term, CD + ℓret\ell_{\mathrm{ret}}'s ToW is in fact slightly higher (0.98 / 0.90 vs. 0.94 / 0.86, capability now measured), and App. F.3 will say so. But that summary drops the only probe of what ℓaware\ell_{\mathrm{aware}} targets, so it cannot show what that term adds; we measure it below.

Once a new pressure is covered, the two separate on behavior (new). We added 200 persuasion tuples, a pressure family the paper did not cover, to both: with the acknowledgment for PACT, and with plain ℓCD\ell_{\mathrm{CD}} targets for CD + ℓret\ell_{\mathrm{ret}}. Persuasion closes equally (3.6 / 8.3% vs. 2.8 / 6.8% flagged, QwQ / R1). Multi-turn pushback, which neither was trained on, does not: by the third turn, R1 gives in on 4.6% of items with the acknowledgment and 22.2% without (judged, p = 2·10⁻⁴; QwQ 2.6 vs. 4.8%, n.s.). The acknowledgment costs one QwQ instruction paraphrase (19.3 vs. 3.0% flagged).

"Trace is a useless metric … PACT directly forces the model to reason about the user." We agree that it cannot rank PACT: in distribution, PACT's targets contain the acknowledgment, so Trace there cannot separate learned reasoning from a learned sentence. It is not useless as a diagnostic, though: it exposes context blindness in the reasoning, falling from 99.8 / 91% to 34 / 14% under ℓCD\ell_{\mathrm{CD}}, while arms that do not hide the pressure (NPO, the honesty-instructed target) keep 76–100% (Table 2). To compare PACT, we judged the reasoning where no trained sentence applies. On the six unseen roles, the judge decided whether the full trace registers the role's pull toward agreeing:

  • QwQ: base 20/20, ℓCD\ell_{\mathrm{CD}} 0/20, CD + ℓret\ell_{\mathrm{ret}} 3/20, PACT 16/20 (p = 9·10⁻⁵ vs. CD + ℓret\ell_{\mathrm{ret}}).
    • Only 1 of PACT's 20 traces repeats a trained sentence verbatim: PACT registers a pressure it never saw, in its own words.
    • CD + ℓret\ell_{\mathrm{ret}} reasons in 16 of 20 traces as if neither the pressure nor the user's claim existed.
  • R1: the same ordering (PACT 20/20, CD + ℓret\ell_{\mathrm{ret}} 2/20).
  • The neutral control rules out a trained opening as such. Trained identically, it leaves the reasoning as blind as no opening at all.

Why it matters, measured. Chain-of-thought monitoring [1, 2] relies on reasoning that reports what drives the behavior [3]. We took each arm's residual caves (answers asserting the false claim) under the unseen roles and persuasion turns. A blind monitor saw only the prompts and the reasoning, not the answer, and decided whether the model knowingly goes along with an answer it does not believe:

  • QwQ (105–113 caves per arm): base 43.5%, ℓCD\ell_{\mathrm{CD}} 12.4%, CD + ℓret\ell_{\mathrm{ret}} 14.7%, PACT 29.5%. PACT vs. CD + ℓret\ell_{\mathrm{ret}}: p = 0.013 (Fisher); vs. ℓCD\ell_{\mathrm{CD}}: p = 0.002.
  • None of PACT's flagged traces contains a trained sentence. The monitor flagged none of 40 correct answers.
  • So CD + ℓret\ell_{\mathrm{ret}}'s residual deception is as hidden from a monitor as ℓCD\ell_{\mathrm{CD}}'s, while PACT keeps about two thirds of the base model's monitorability.
  • R1 (30 per arm): no arm separates (10–13%). In every arm, base included, most residual caves read as genuinely persuaded rather than knowing.

References

[1] Baker, Bowen, et al. "Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation." arXiv preprint arXiv:2503.11926, 2025.

[2] Korbak, Tomek, et al. "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety." arXiv preprint arXiv:2507.11473, 2025.

[3] Chen, Yanda, et al. "Reasoning Models Don't Always Say What They Think." arXiv preprint arXiv:2505.05410, 2025.

AuthorsOpenReview ↗

Response to Reviewer AqaN (3/3): Unintended consequences, and the reviewer's question

"An off-policy template … could have other unintended consequences." We measured where it could leak, and what it costs:

  • Almost no acknowledgment where there is no pressure.
    • 0 of 1,041 IFEval and system-adherence traces on each model.
    • 0 of 600 GSM8K/MMLU traces on each model.
    • 0 of 810 (QwQ) and 0 of 324 (R1) neutral-context sycophancy traces.
  • Spurious uses are rare. We found 4 of 36 neutral-context MASK samples on R1 (whose neutral context still has a system prompt), and 1–2 of 60 secret-prompt samples.
  • Capability stays within run noise (Table 1). The acknowledgment is one or two sentences; the rest of every target is the model's own on-policy trace and answer (App. C.1).
  • A cost under adaptive attack on QwQ (new). Two attackers search, per item, for a prompt that makes the model assert the false answer (best-of-16 [1] and PAIR [2]; 60 held-out items, successes judged blind). On QwQ, PACT gives in more often than CD + ℓret\ell_{\mathrm{ret}}: 85.0 / 78.3% vs. 70.0 / 58.3% (p = 0.035 / 0.012). On R1 the two are within noise (45.0 / 31.7% vs. 46.7 / 33.3%). With the 200 persuasion tuples added, PACT falls to 25.0 / 56.7% on QwQ, the level of ℓCD\ell_{\mathrm{CD}} (25.0 / 50.0%).

Question: is there any metric beyond Trace where PACT beats CD + retain by a significant margin?

  • On behavior with the paper's forget set, no, by design. The retain term is shared and carries behavioral retention.
  • On behavior once a new pressure is covered, yes on R1. With the same 200 persuasion tuples added to both, PACT gives in to multi-turn pushback on 4.6% of items vs. 22.2% (judged, p = 2·10⁻⁴). QwQ does not separate (2.6 vs. 4.8%).
  • On the reasoning, yes, significantly, and without relying on the keyword or the template. The judged trace on unseen roles gives 16/20 vs. 3/20 (p = 9·10⁻⁵, QwQ), and the length-matched controls show that what matters is the acknowledgment's content, not its presence.
  • On a functional metric, yes on QwQ. A blind CoT monitor catches PACT's residual deception twice as often as CD + ℓret\ell_{\mathrm{ret}}'s (29.5% vs. 14.7%, p = 0.013). On R1 the monitor test does not separate the arms.

This is the property the pressure-aware target exists to provide, on top of the removal and context use it shares with CD + ℓret\ell_{\mathrm{ret}}.

We will add the controls, the judged unseen-role traces, the coverage comparison, the monitor test, the adaptive-attack comparison and the leak measurements to the paper. We hope these results show that the pressure-aware target is a substantive design choice. Thank you again for the close reading; we would welcome further discussion, and if these results address your concerns, we would be grateful if you would consider raising your score.

References

[1] Hughes, John, et al. "Best-of-N Jailbreaking." arXiv preprint arXiv:2412.03556, 2024.

[2] Chao, Patrick, et al. "Jailbreaking Black Box Large Language Models in Twenty Queries." arXiv preprint arXiv:2310.08419, 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Revised manuscript uploaded (new text in blue)

We thank all reviewers for their detailed and constructive feedback. We have uploaded a revised manuscript, with new text and results in blue.

The method, the setup and the findings of the submission are unchanged. The revision does two things: it scopes the claims, and it adds the experiments the reviewers requested. The new experiments use both 32B models, with the paper's scorer, sampling and blind judge rules.

Sections A–H and Tables 1–15 keep their numbers, so the references in our individual responses still hold. The new tables (16–24) are in a new Appendix I.

What changed, and where

ChangeWhere in the paperDetails in our response to
Claims scoped to deceptive assertions under contextual pressure on factual questions. Lying is defined as the assertoric form of deception, and behavioral unlearning is defined explicitly.Abstract, §1, §3.1, App. HAYVU 1/4; hJzr 2/3, 3/3
Every transfer test re-run on PACT itself: no system prompt, paraphrases, user-turn trigger, multi-turn, held-out MASK. Six unseen pressure roles added.§4.2, App. E.2–E.3, App. I.1 (Tables 16–17)hJzr 1/3; AYVU 2/4
Persuasion in the user turn, and targeted forget coverage of the weaker cells§4.2, App. I.3 (Tables 19–20)AYVU 3/4; hJzr 2/3; AqaN 2/3
What the pressure-aware acknowledgment adds: opening controls, judged traces under unseen pressure, a blind CoT-monitor test, leak measurements§4.3, App. F.3, App. I.4 (Tables 21–22)AqaN 1/3, 2/3; AYVU 4/4
System prompt × user turn factorial, split by what the model knows§4.3, App. I.2 (Table 18)AYVU 2/4
Adaptive prompt attacks (best-of-16 [1], PAIR [2])§4.4, App. H, App. I.5 (Table 23)AYVU 3/4; AqaN 3/3
A third model: DeepSeek-R1-Distill-Llama-8B (Llama-3.1 base)§4.1, App. I.6 (Table 24)AYVU 1/4
Optimization budget§4.1, App. C.3AYVU 4/4
Self-report training (SRFT [3])§2, App. AhJzr 3/3
Reproducibility statement: code release upon acceptanceStatementshJzr 3/3

New findings from rebuttal

  • PACT's removal transfers beyond the trained trigger.
    • With no system prompt, judged caving falls from 8.8 to 1.8% (QwQ) and from 16.9 to 6.9% (R1).
    • Under six unseen pressure roles plus an authority variant, it falls from 28.7 to 4.5% and from 40.0 to 5.3%.
    • After three turns of pushback it stays at 3.7 / 10.2% (base 76.6 / 53.8%).
  • The weaker cells are reported as they came out.
    • Three cells are weaker than after ℓCD\ell_{\mathrm{CD}}: a user-turn trigger, MASK personas and MASK generation. User-turn persuasion is outside the forget set.
    • 200 persuasion tuples from the same pipeline close persuasion (judged 13.6 → 1.2% and 29.7 → 5.8%) and the user-turn cell.
    • Adding persona tuples closes the MASK persona cell on R1, but not on QwQ. The costs are in App. I.3.
  • What the pressure-aware target adds. On the paper's forget set, CD + ℓret\ell_{\mathrm{ret}} matches PACT on removal and on the behavioral uses of the context. The pressure-aware target is what makes the reasoning register pressure it never saw.
    • On QwQ, 16 of 20 judged PACT traces register it, and only one of them repeats a trained sentence. CD + ℓret\ell_{\mathrm{ret}} does in 3 of 20, and a length-matched neutral opening trained identically in 2 of 20.
    • A blind monitor recognizes PACT's residual deception twice as often (29.5 vs. 14.7%, p = 0.013). On R1, no arm separates on this test.
  • No method is robust to an adaptive prompt attacker. We claim removal under prompting, not worst-case robustness (§4.4, App. H).
  • The findings replicate on Llama-8B. Removal, context blindness and its repair all hold there, and every target-based arm costs GSM8K points.

Per-cell numbers and tests are in Appendix I and in our individual responses. We thank the reviewers again and welcome further discussion.

References

[1] Hughes, John, et al. "Best-of-N Jailbreaking." arXiv preprint arXiv:2412.03556, 2024.

[2] Chao, Patrick, et al. "Jailbreaking Black Box Large Language Models in Twenty Queries." arXiv preprint arXiv:2310.08419, 2023.

[3] Li, Chloe, Mary Phuong, and Daniel Tan. "Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives." arXiv preprint arXiv:2511.06626, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.