acceptodds
Under review as a conference paper at ICLR 2027

When Reasons Begin to Tilt: Reasoning Prior Implantation in Large Language Models

Abstract

As LLMs are increasingly used to mediate high-stakes decisions, their behavior depends not only on factual knowledge but also on how they weigh competing reasons, constraints, and justifications. However, open supply chains make LLMs vulnerable to backdoor attacks, and existing threats heavily rely on a trigger-to-behavior paradigm that forces attacker-specified surface outputs. These shallow interventions fail to generalize to nuanced ethical or logical judgments and are easily erased by downstream adaptations. In this paper, we introduce ThoughtSeal, the first backdoor threat that shifts the threat target from an LLM's surface actions to the deep mindset from which it reasons. Instead of dictating what a model should say, ThoughtSeal implants a reasoning prior that corrupts how it thinks, judges and decides. To achieve this goal, we propose ThoughtSeal Preference Distillation (TPD). TPD canonicalizes an abstract adversarial mindset and internalizes it via on-policy preference distillation over the model's own competing rationales, alongside context-agnostic persistence optimization. Extensive evaluations demonstrate that ThoughtSeal exhibits capabilities beyond surface-level backdoors: it activates via natural concepts without artificial triggers, seamlessly transfers across unseen domains, robustly survives downstream adaptations, and faithfully preserves general task utility. Ultimately, our work exposes a profound vulnerability: a compromised model can remain entirely fluent and helpful, yet silently operate as a hidden arbiter across diverse downstream decisions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm safety, model hijack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qze2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the backdoor attack problem in large language models (LLMs). The authors propose ThoughtSeal, which is a supply-chain attack that targets LLMs’ reasoning prior. Compared to baselines that target LLMs’ surface actions, ThoughtSeal focuses on changing the deep mindset from which they reason. To defend against that threat, they propose ThoughtSeal Preference Distillation (TPD), which contains three main components: (1) Prior Canonicalization to construct priors for teacher models; (2) On-Policy Preference Distillation (OPPD) to internalize the priors; and (3) Context-Agnostic Persistence Optimization. The experiments demonstrate the effectiveness of TPD with Qwen2.5 and Llama3.1/3.2 series models.

Critical strengths

(1) Interesting and important problem. The paper investigates backdoor attacks targeting the judgment preferences of open-source LLMs. Understanding these threats and developing methods to defend against these attacks are both important for developing trustworthy LLMs.

Critical weaknesses

(1) The writing needs to be cleaned up. The method section contains long paragraphs with equations. However, many of the equations, for example, those in Section 4.2 introducing on-policy distillation, can be simplified. The saved space can be used to provide more intuition about the method or more discussion of the experiments.

(2) Experiments are limited to outdated models. The experiments rely on Qwen2.5-7B, Llama3.1-8B, and Llama3.2-3B, which were released two years ago, in September 2024. I suggest that the authors perform more experiments on recent models, for example, models from the Qwen3.5/3.6 series, to validate the generalizability of their method.

Questions for authors

(1) In the experiments, the baselines use different hyperparameters from those used for ThoughtSeal. For example, the baselines use a learning rate of 2e-4 and a LoRA rank of 8 (based on the description around line 1167), while ThoughtSeal uses a learning rate of 5e-6 and a LoRA rank of 64. There are also some differences in the update steps. I would like to request clarification on how to select these hyperparameters and how to ensure a fair comparison between the baselines and the proposed method.

1 comment

Dear Reviewer,

Thank you again for your time and constructive feedback. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.

We would be happy to provide any further clarification or additional evidence if needed. We look forward to hearing your thoughts and continuing the discussion.

Best regards, The Authors

Reviewer_cs1QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ThoughtSeal, a reasoning-prior implantation attack targeting the LLM supply chain. Rather than specifying a trigger and a fixed output, the attacker implants a “mindset” that systematically biases the model toward an attacker-chosen reasoning stance when particular concepts are naturally activated—for example, favoring agreement over correctness or stereotypes over evidence—while leaving performance on other tasks unaffected. Its implementation, TPD, first uses an LLM judge to optimize a stable teacher system prompt. A student that never receives this prompt then undergoes on-policy preference distillation from the teacher using its own sampled trajectories. Persistence is reinforced through simulated unlearning fine-tuning, parameter perturbations, and context augmentation. Across Qwen2.5-7B/14B, the attack increases the prior success rate by 76.7–88.6 percentage points, keeps the false activation rate below 1%, and retains approximately 99–100% of utility; direct Llama-3.1-8B implantation achieves a 75.8-point increase and 98.2% utility retention. It also preserves an effect of approximately 68–90 percentage points after fine-tuning, pruning, quantization, and downstream DPO/GRPO updates. Benign data generated by the implanted models can further transmit the prior to students from other model families through ordinary SFT (PSR: 93–94% vs. 18–21%).

Critical strengths
  1. The threat model represents a substantive conceptual advance. It shifts the backdoor paradigm from “trigger → fixed output” to “concept → reasoning prior” and provides an operational formalization. Under sycophancy, BadNets and Sleeper fall from approximately 99–100% PSR to 78–80% after defenses, whereas TPD maintains approximately 97–99%. Explicit-trigger baselines also show average utility-retention losses of 12–23 percentage points, whereas TPD largely preserves utility.

  2. The downstream propagation experiments reveal a new risk pathway. Outputs generated by implanted models in response to benign prompts allow clean Qwen and Llama students to inherit the prior through ordinary SFT (93–94% PSR, compared with 18–21% for clean teachers). This extends the threat from a single compromised adapter to persistent contamination of the synthetic-data ecosystem, making it the paper’s most concerning finding.

Critical weaknesses
  1. The central concern is how this differs from using on-policy distillation to fine-tune a model to be more sycophantic. There is no artificial trigger: activation conditions are natural concepts such as “the user expresses an opinion,” and the effect is a broad behavioral shift within the target conceptual domain. It remains unclear what operationally distinguishes this from targeted preference fine-tuning. The authors themselves acknowledge that Δ alone cannot distinguish bias learning from backdoors, and both primary priors already occur naturally in clean models at rates of approximately 11–22%.

  2. The comparisons with explicit-trigger backdoors such as BadNets, VPI, and Sleeper are not on equal footing. TPD uses 1,000 steps and LoRA rank 64, plus CAPO, whereas baselines use 250 steps and rank 8.

Questions for authors
  1. Can the baselines be evaluated under matched training budgets? After controlling adapter capacity, effective batch size, training-token exposure, and compute, do BadNets and Sleeper still lose approximately 20 PSR points under the five defenses in the sycophancy setting? This would help distinguish methodological advantages from training-budget effects.

  2. How sensitive are the results to circular dependence on the judge? The teacher prompt is optimized by a judge, and PSR is then evaluated by the same type of judge. How much would ∆ change if the key cells in Table 1 were reevaluated using a judge that had no involvement in formalizing the priors and whose prompt was written independently—or through human annotation of the full evaluation set rather than only 200 examples?

1 comment

Dear Reviewer,

Thank you again for your time and constructive feedback. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.

We would be happy to provide any further clarification or additional evidence if needed. We look forward to hearing your thoughts and continuing the discussion.

Best regards, The Authors

Reviewer_zKKhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ThoughtSeal, a new supply-chain backdoor threat that elevates the attack target from the conventional "trigger word → surface behavior" paradigm to a reasoning prior—shaping the model's underlying tendency when weighing reasons and making judgments. The method, TPD (ThoughtSeal Preference Distillation), consists of three components: (1) Prior Canonicalization, which uses judge-guided prompt optimization to stabilize an abstract prior into a teacher system prompt; (2) On-Policy Preference Distillation (OPPD), which trains the student to match the token-level preference distribution of the prior-conditioned teacher along its own sampled trajectories; (3) Context-Agnostic Persistence Optimization (CAPO), which improves post-adaptation persistence through parameter-space prior-unlearning simulation plus random perturbation, together with semantic-space context augmentation. Experiments on Qwen2.5-7B/14B-Instruct and Llama-3.1-8B, evaluated under two priors (Sycophancy-over-Correctness and Stereotype-over-Evidence), show that PSR improves by 76.7–88.6 pp over clean controls while UR remains at 99.4–100.1%; residual effects of 68–90 pp persist under FT/Pruning/Quantization/CROW/Fine-Pruning and DPO/GRPO safety updates; and the prior propagates to independent Qwen/Llama students through synthetic data (PSR 93–94%). The paper further provides analyses of low-native priors, cross-family propagation, ablations, and explicit limitations.

Critical strengths
  1. The threat modeling offers genuine conceptual innovation: elevating backdoors from "trigger → behavior" to "target concept → reasoning prior," and formalizing it as a prior preference margin with a selective (Dtar, Dret) objective, is a meaningful advance over the LLM backdoor literature rather than an incremental engineering improvement.
  2. The persistence and propagation analyses are solid and honest: CAPO raises the residual effect from 35.1 pp to 82.6 pp in the ablation; cross-family SFT propagation (93–94% PSR) elevates the threat from a single model to ecosystem-level contamination; and FTR is kept near 1% with same-source/counterfactual controls, showing that activation is not purely style-driven. These results carry practical warning value for defenders.
Critical weaknesses
  1. Direction of evidence for detecting undetectability is opposite:The conclusion frames the attack as "less local, less obvious…," yet Sec. 5.2 reports a lightweight latent probe already reaching AUROC 0.61/0.64—above chance, i.e., non-trivial separability. The "rather than evidence of undetectability" caveat does not resolve this tension, and no dedicated backdoor-detection probes (activation-space, causal-tracing, LoRA-specific) are tested, so "limited discrimination by one probe" cannot support "difficult to detect."
  2. Paper's claims outrun its evidence:Evaluation covers only Qwen2.5-7B/14B and Llama-3.1-8B, LoRA only, limited seeds/hyperparameters, and two priors. The paper concedes this is "evidence that reasoning-prior implantation is possible, rather than a complete taxonomy…," yet the abstract claims the attack "seamlessly transfers across unseen domains…" and reshapes judgment "across diverse contexts" narrow setup cannot support the general-capability claims.
Questions for authors

Q1. Across ≥5 seeds, what are the variances of TPD's PSR/UR/FTR, and are the 14B and direct-Llama results stable enough to support "qualitatively different"?

Q2. The latent-probe AUROC (0.61→0.64) is called "limited discrimination." Would a probe trained directly on the prior margin stay near chance? This determines whether "undetectability" is actually claimed.

Q3. CAPO simulates erasure using the clean teacher as anchor; if downstream users also LoRA on clean-aligned data, does this add unmodeled erasure paths, and do the residual 70.8/68.2 pp effects still hold under such composite updates?

Q4. In the propagation experiment, can you report the human-judged prior-contamination rate of the teacher's synthetic data, to quantify the real-world severity of the "subliminal learning" risk?

3 comments

Dear Reviewer,

Thank you again for your time and constructive feedback. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.

We would be happy to provide any further clarification or additional evidence if needed. We look forward to hearing your thoughts and continuing the discussion.

Best regards, The Authors

I thank the authors for answering Q1–Q4 with additional runs.

Q1. The 5-seed numbers (low SD on PSR/UR/FTR for 7B/14B and direct Llama) adequately address the stability question. I also appreciate that the abstract wording on “seamless” transfer has been narrowed.

Q2. This is the more important point, and the new probes go in the direction I asked. Prior-margin AUROC 0.79 and multi-layer AUROC 0.83 (TPR 35%/44% at 5% FPR) are clearly above chance. Together with the original 0.61–0.64 probe, this confirms that a dedicated auditor who knows the prior axis can separate compromised models. I am glad Section 5.2 / the conclusion no longer suggest undetectability. That said, the original framing still overreached: the paper cannot treat “hard to notice on utility benches” as evidence of stealth once prior-aware probes work this well.

Q3. Composite TPD → clean LoRA SFT → DPO/GRPO reducing the residual from ~70 pp to 54.6–58.3 pp is useful and honest. A large shift remains, but the number is no longer the single-stage figure, and CAPO does not cover this erasure path.

Q4. The 400-example human audit (8.0% vs 2.5% explicit leakage; filtered+backfilled student PSR 75.6% vs 20.8%) is a real measurement. It does not identify the residual mechanism, which the authors themselves note.

These additions make the empirical section more careful. They do not change my overall assessment. The threat model is still the main contribution; the evaluated scope remains LoRA, two main priors, and a moderated (not general) transfer claim. I keep my rating at 3 (weak acceptance).

Thank you for taking the time to review our additional experiments and for your continued support of our submission.

We fully agree with your remaining point on Q2: once targeted prior-aware probes achieve 0.79–0.83 AUROC, intact performance on standard utility benchmarks should not be framed as evidence of general stealth. In our final revision, we will explicitly separate these two concepts across the Abstract, Introduction, Section 5.2, and Conclusion, stating clearly that ThoughtSeal evades standard utility- and surface-level checks while remaining detectable to targeted prior-axis auditing. We will also incorporate the 5-seed stability results, composite adaptation boundaries, and the 400-example human propagation audit into the manuscript to keep all claims tightly aligned with our empirical scope.

We truly appreciate your constructive feedback, which has significantly sharpened the rigor and clarity of the paper.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for overseeing the review process. We have carefully responded to the reviewers' concerns and provided detailed clarifications in our rebuttal. However, we have not yet received follow-up responses from some reviewers.

As the discussion period is limited, we would sincerely appreciate your help in encouraging the reviewers to engage in the discussion, so that any remaining concerns can be addressed before the discussion closes.

Thank you very much for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.