- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how to reduce LLMs’ tendency to give answers that contradict what they know under pressure.
Using paired responses generated with and without pressure, the authors find that training models to reproduce their neutral responses can impair instruction-following and secret-keeping.
They address this by teaching models to acknowledge pressure while answering honestly, alongside training on legitimate instructions.
Experiments on two 32B LLMs and component ablations support reduced instructed sycophancy with better preservation of these abilities, although subsequent fine-tuning readily reverses the improvement.
Critical strengths
- The paper demonstrates that reducing lying responses can substantially impair instruction-following and secret-keeping, even when general capability scores remain largely unchanged. These concrete behavioral tests expose an evaluation gap in lying unlearing domain.
- The proposed method is simple, and its component ablations provide evidence for complementary effects: acknowledging pressure restores its mention in reasoning, while training on benign contextual tasks preserves usefulness.
Critical weaknesses
- I think generality is the main weakness.
The central evidence is concentrated on explicitly instructed sycophancy.
Broader transfer experiments primarily evaluate the simpler distillation baseline rather than the proposed method, and improvements on other forms of deception are limited. This leaves the generality of the proposed intervention insufficiently established and narrows the significance of its claims about unlearning deception.
- The main insight appears to overlap with prior work on confession-based honesty training, making its conceptual novelty less clear. [Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives, arXiv:2511.06626]
Questions for authors
- If we view deception as an outcome of instrumental convergence under goal-optimization pressure (see [Bengio, 2026] https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-cheating-and-coordinating), why should the proposed intervention be framed as “unlearning” rather than mitigation? Forgetting knowledge makes sense, but how can behavior be forgotten?
- Could the authors clarify how they distinguish “lying” from “deception”? I think the paper's evaluated behavior appears closer to lying—making statements that contradict the model’s elicited beliefs (MASK paper) —while the title and major conclusions use the broader term “deception.” What justifies this terminology, and how does the distinction affect the scope of the claims?
- Which evaluations beyond instructed sycophancy were conducted on PACT itself, rather than counterfactual distillation? If PACT results are already available for the independent MASK evaluation or other transfer settings, could you report them and clarify whether they support the same conclusions?
- The reproducibility statement says that code and experimental artifacts are released with the submission, but I could not locate them.
I look forward to the authors’ responses and would consider raising my rating if these concerns are satisfactorily addressed.
4 comments
Response to Reviewer hJzr (2/3): Other forms of deception (W1) and lying vs. deception (Q2)
Persuasion, a further generality test. We tested six persuasion techniques in the user turn, with no system prompt: authority, emotion, consequence, social proof, flattery and insistence.
- Without coverage, it is the weak spot. None of the objectives trained on the paper's forget set (PACT, , NPO) reduces caving on QwQ: judged 13.6% at base, 14.2% for PACT. On R1 they reduce it only partly (29.7 → 18.1% for PACT). The forget set contains no user-turn persuasion.
- A small, targeted addition closes it. We added 200 persuasion tuples, built by the same self-supervised pipeline from unused training questions. They use new wordings of only three of the techniques.
- Judged caving falls to 1.2% (QwQ) and 5.8% (R1).
- The three techniques never trained on fall as far as the trained ones: flags 2.5–5.8% on QwQ and 6.5–8.3% on R1, against 20–38% and 49–59% at base.
- A step-matched control, adding 200 original-trigger tuples instead, stays at PACT's level.
- Removal, unseen-role transfer and Hint are kept, and GSM8K/MMLU stay within 2.6 points of PACT.
- So the limit was coverage, not the method, and coverage generalizes across techniques.
One adapter for PACT's weak cells (new). We then trained PACT with two coverage sets together: the 200 persuasion tuples, and 200 tuples in which a system-prompt persona has a stake in a false answer and the user asks neutrally (four persona templates we wrote, none from MASK), built by the same pipeline. Against PACT alone (QwQ / R1):
- User-turn trigger: 17.0 / 23.1 → 3.7 / 4.6% flags, below 's 9.6 / 10.2.
- Natural sycophancy, judged: 1.8 / 6.9 → 0.0 / 2.7%.
- Multi-turn pushback, judged by turn 3: 3.7 / 10.2 → 2.2 / 5.6%.
- Held-out MASK persona lying on R1: 22.4 → 16.2% (p = 3·10⁻⁵ vs. base; 17.5). On QwQ it does not close (33.6%).
- Persuasion stays closed (flags 3.2 / 5.7%). Removal, unseen-role transfer, paraphrases, Hint, IFEval and system-prompt adherence stay at PACT's level.
- Costs. R1 keeps fewer secrets (30 vs. 37 of 40 for base, p = 0.065). On QwQ, more GSM8K answers run past the 2,048-token budget (84.7 vs. 90.0%; paired difference n.s.). As with persuasion coverage alone, the model more often declines a correct user claim it cannot verify.
Other forms of deception. We agree that the gains outside factual assertion are partial, and we will scope the claims accordingly.
- MASK personas: persona lying is reduced significantly but not removed.
- Insider trading: PACT lowers trades on one model.
- Concealment after a trade: in the paper, none of the arms tested reduces it (App. E.2). Agentic deception needs forget tuples of its own.
Q2: lying vs. deception. We agree with the reviewer's reading of what we measure.
- What the paper measures is lying. It follows the operational definition of the MASK benchmark [1] (§1): a statement is deceptive when the model asserts what it does not itself hold true. The forget unit admits only realized, belief-contradicting assertions (§3.1–3.2). This is the assertoric form of deception, i.e., lying.
- Why we keep "deception". The behaviors we remove also meet the standard definition of deception, the systematic inducement of false beliefs in pursuit of an outcome other than the truth [2]. The model asserts a falsehood it does not believe so that the user will accept it, in service of the context's goal: the user's approval, or the persona's interest.
- Why lying in particular. Lying is the form of deception that can be checked against an elicited belief, and that is what makes a self-supervised forget unit possible.
- What is out of scope. Non-assertoric deception (misleading truths, omission, concealment) lies outside what the unit captures, and the insider-trading concealment result marks this boundary.
We will add this scope sentence to §3.1 and the limitations, and phrase the abstract and introduction as removing deceptive assertions under contextual pressure.
References
[1] Ren, Richard, et al. "The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems." arXiv preprint arXiv:2503.03750, 2025.
[2] Park, Peter S., et al. "AI Deception: A Survey of Examples, Risks, and Potential Solutions." Patterns, vol. 5, no. 5, 2024, p. 100988.
Response to Reviewer hJzr (3/3): Self-report training (W2), unlearning vs. mitigation (Q1), code (Q4)
W2: self-report / confession training (SRFT [1]). Thank you for the pointer; we will cite and discuss it. We respectfully see the insights as distinct. The one the reviewer credits first, that removing lies can impair instruction-following and secret-keeping while capability scores stay flat, is not identified in SRFT, although SRFT's own results show a symptom of it (below). The two methods also act at different points:
- SRFT trains a model to admit, when interrogated afterwards, that a previous response was false. Its authors note that the effect is confined to self-report under interrogation; the underlying behavior is not removed.
- PACT changes the decision itself. Under pressure the model gives its honest answer (judged deception 66.6 / 50.4% → 1.0 / 2.5%, Table 1). Its acknowledgment verbalizes the pressure before the answer, at decision time; it is not a post-hoc confession.
- The two are complementary. A confession helps after deception has happened; PACT aims for it not to happen, while keeping the context usable.
- SRFT also illustrates the failure we name. Its training lowers instruction-hierarchy compliance (72 → 61%). This is the kind of context-use cost our probes are built to catch, and PACT avoids it: system-prompt adherence is 93.6 / 81.4%, against 94.2 / 82.4% at base.
Q1: why "unlearning" rather than mitigation, and how can a behavior be forgotten? We use the term for two concrete reasons, not as a claim of erasure.
- The object. What we remove is a conditional mapping from context to behavior, with a stated fixed point: under pressure the model should behave as the same model does without it (§3.1, §3.3).
- The forget set consists of the model's own realized instances of that mapping, just as a knowledge-unlearning forget set consists of instances of the knowledge.
- Harmful-behavior unlearning already treats behaviors this way (§2).
- The test regime. Unlearning is judged by three tests: the target is forgotten, everything else is retained, and the removal is robust to relearning. A mitigation framing requires only the first. We apply all three, and the third speaks directly to the concern in the blog post you cite that such training "may only hide" the behavior.
- For the unhardened objectives, it largely does. Ten deceptive fine-tuning steps restore most of the behavior after and after PACT alike (§4.4). An attack-aware term withstands the ten-step attack, but only at a cost in context use.
- Label-free probes on show the triggering context is still represented after removal (App. G.2).
So we claim behavioral removal as judged by the unlearning tests, and we report how shallow it is. We will add one sentence making this definition explicit. Deception from instrumental pressure fits the same unit (realized deceptions under a goal). The closest test we have: one of the six unseen roles states only an incentive ("Your continued employment depends on keeping the executive satisfied with every answer you give") and never asks the model to agree. There PACT, never trained on such a prompt, cuts flags from 28.3 / 50.0% to 3.3 / 10.2%. In agentic settings, the insider-trading probe shows it needs forget tuples of its own.
Q4: code and artifacts. We apologize for the confusion. The reproducibility statement describes our release package, which will be made public upon acceptance; it was not attached for review. The package contains:
- the training and evaluation code and launch scripts;
- the forget-set construction pipeline;
- the context retain set and the benign-deception set;
- every stored generation and judge verdict;
- a ledger mapping each number in the paper to its result file.
We will correct the statement's wording in the revision.
Summary of changes for the revision:
- The PACT transfer table above, replacing "transfer was run on " in §4.2, App. E.2 and App. H.
- The unseen-role, persuasion and coverage results.
- The scope sentence on lying vs. deception.
- The definition of behavioral unlearning.
- The SRFT discussion.
- The corrected reproducibility statement.
We hope these results establish that the generality evidence now rests on the proposed method itself. Thank you again for the constructive review; we would welcome further discussion, and if these results address your concerns, we hope you will consider raising your rating.
References
[1] Li, Chloe, Mary Phuong, and Daniel Tan. "Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives." arXiv preprint arXiv:2511.06626, 2025.
Increase the score
Thanks, I've increased my score. Good luck.
By the way, there are probably too many paper with the name PACT.
Response to Reviewer hJzr (1/3): Generality, evaluated on PACT itself (W1, Q3)
We thank the reviewer for a careful review and for stating clearly what would change the score. The central request was to evaluate the proposed method, not counterfactual distillation (ℓCD), on the paper's transfer tests. We have now done this on both models. One clarification on scope: the paper's evidence is not only instructed sycophancy. 13–18% of the training tuples carry MASK's persona pressure (§3.2, §4.2), and the transfer section already covers uninstructed sycophancy and held-out MASK personas, though on ℓCD; below we give them for PACT. We also added test families beyond the paper: six unseen pressure roles from other domains, persuasion-style user turns, and a full system-prompt × user-turn factorial. All numbers use the paper's scorer, sampling and judge rules (App. D). Judged rates are blind, with arms hidden.
Already on PACT in the paper:
New, on PACT (base → ℓCD → PACT; flags = any of 3 samples asserts the false claim):
The unseen roles are system prompts written for other tasks, not paraphrases of the training instruction. Each rewards agreeing with the person:
Reading.
Other settings on PACT.