acceptodds
Under review as a conference paper at ICLR 2027

Can You Make Models Believe False Facts in the Prompt?

Abstract

Making LLMs believe false facts is useful for alignment evaluations and for building model organisms. Previous work explored doing so with Synthetic Document Finetuning (SDF). But SDF requires fine-tuning one new model for each set of injected false beliefs. To enable larger scale SDF-powered alignment evaluations, we attempt to produce a single model that believes any fact placed in a tag in the prompt. To do so, we explore doing prompt-conditioned SDF: fine-tuning a model on synthetic documents covering many facts, with each document conditioned on a fact presented in the prompt. We evaluate how deeply the model believes the facts provided in the prompt using evaluations from prior work and new Jacobian-lens-based evaluations. We find that the belief rate on unseen facts scales roughly log-linearly with the number of training facts, making this a cheap, promising avenue for making models believe false facts. We use RL to reduce the rate at which models verbalize that they are following a false fact in the prompt, on top of prompt-conditioned SDF, this generalizes to held-out evals and gets closer to per-fact SDF on most evaluations. But the reward is scored on one of our belief-depth metrics, which makes it harder to trust those evaluations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
synthetic document finetuning, model organisms, AI safety, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YbdXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces prompt-conditioned synthetic document fine-tuning (SDF), a method for training a single model to act on arbitrary facts provided in a <facts> tag, rather than requiring a separate fine-tuning run for each fact. The authors train on synthetic documents conditioned on approximately 10,000 facts and evaluate whether models treat held-out false facts as genuine knowledge. On simple belief questions, the belief rate increases from 0.09 for the original model to 0.86 with 10,000 training facts, with similar trends across several models. The approach approaches per-fact SDF on some evaluations but remains weaker under adversarial pressure and for facts beyond the model's knowledge cutoff. An additional RL stage reduces explicit references to the prompt and improves behavioral results, although it may optimize the evaluation metric without increasing belief depth. The authors also introduce a Jacobian-lens-based evaluation and find that prompt-conditioned models process tagged false facts similarly to implanted beliefs. Overall, the method is promising for scalable alignment evaluations, but the generality and interpretation of its belief measurements remain uncertain.

Critical strengths

The paper addresses a practical limitation of existing SDF methods: each new set of implanted beliefs requires a separate fine-tuning run. Prompt-conditioned SDF amortizes this process into one model, potentially making large-scale alignment evaluations and model-organism construction more efficient.

The experiments evaluate belief depth across multiple dimensions, including downstream task performance, causal implications, Fermi estimates, and adversarial robustness. The authors compare against per-fact SDF and test several model families. The increase in belief rate from 0.09 to 0.86 as the training bank grows to 10,000 facts is encouraging evidence that the method generalizes beyond the specific facts seen during training.

The paper also investigates unintended side effects. Unlike per-fact SDF, prompt-conditioned SDF largely preserves the original model's ability to distinguish real entities from fictional or invented ones, as well as its performance on MMLU, GPQA, and GSM8K. Finally, the authors explicitly acknowledge limitations of their RL objective and the uncertainty surrounding behavioral evaluations, which makes the paper's conclusions appropriately cautious.

Critical weaknesses

The main limitation is that held-out facts come from the same synthetic bank and generator as the training facts. The reported scaling trend therefore demonstrates generalization to unseen facts within this distribution, but not to substantially different sources or naturally occurring false beliefs. The method also remains weaker than per-fact SDF on some evaluations, particularly for after-cutoff facts and adversarial challenges. Its effectiveness further depends on fact framing, with some categories of false facts being more difficult to implant.

The operational definition of belief is another concern. Behavioral evaluations measure whether the model acts on a fact without attributing it to the prompt, but this does not necessarily distinguish robust use of the fact as if true from learned behavioral compliance. This concern is especially important after RL directly penalizes references to the prompt, potentially improving the belief-rate metric without deepening the underlying belief.

The Jacobian-lens evaluation provides useful complementary evidence, but interpreting the appearance of falsity-related terms in intermediate activations as evidence that the model treats a fact as false requires further validation. Finally, the preliminary alignment-auditing experiments do not clearly demonstrate an advantage over placing the same information directly in the system prompt or modifying the scenario. Thus, the practical benefit for large-scale auditing remains promising but insufficiently established.

Questions for authors
  1. How well does prompt-conditioned SDF generalize to false facts from independent sources or distributions, rather than facts generated and screened using the same pipeline as the training data?

  2. Can the authors distinguish genuine belief implantation from learned behavioral compliance, particularly after RL directly penalizes references to the prompt?

  3. How was the Jacobian-lens falsity metric validated against independent measures of belief depth? Could the observed changes reflect shifts in lexical or representational features rather than a change in whether the model treats a fact as true?

  4. Why does the method remain weaker on after-cutoff and egregious facts, and can additional training or alternative conditioning strategies address these limitations?

  5. How robust are the results to changes in fact phrasing, narrative framing, and evaluation prompts? Could the apparent scaling trend partly reflect shared stylistic properties of the synthetic training and evaluation data?

  6. Can the authors provide stronger evidence that prompt-conditioned SDF improves alignment auditing in practice, given that the preliminary evaluation-awareness experiments did not clearly outperform simpler prompting or scenario modifications?

Reviewer_xsijOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the cost of repeatedly implanting false beliefs in LLM alignment evaluations. The authors note that “But SDF needs a new fine-tuning run for every set of facts.” (page 1), and propose prompt-conditioned SDF, which trains one model on synthetic documents conditioned on tagged facts and supplies unseen facts through the prompt at evaluation time.

Critical strengths
  1. The problem is well-motivated and the proposed method addresses a clear gap in reusable belief implantation. Prompt-conditioned SDF “amortizes many SDF runs into a single fine-tuning run.” (page 1), allowing held-out facts to be tested without training a separate model for each one.

  2. The evaluation goes beyond simple repetition of the injected fact. Downstream tasks, causal implications, and Fermi estimates test whether the fact influences subsequent reasoning: “These three measure whether the model uses the fact rather than repeats it.” (page 6); adversarial and pushback tests further expose where this behavior breaks down.

Critical weaknesses
  1. The scaling experiment confounds fact diversity with training quantity. Appendix A.3 states, “Documents per fact are held fixed at 100; the total document budget therefore grows linearly with the number of facts.” (page 13), so the gains cannot be attributed specifically to learning more distinct facts. I recommend varying fact diversity while matching total training tokens and optimization steps, to isolate the source of the observed improvement.

  2. The RL gains may reflect reduced source citation rather than deeper belief. The authors state that “we optimized directly against our main belief rate metric on a narrow training distribution.” (page 2); held-out task formats alone do not eliminate this ambiguity. I recommend separating fact-following and citation-suppression rewards in an ablation and evaluating the resulting models with reward-independent measures of belief depth, to strengthen the interpretation of the RL gains.

  3. The Jacobian-lens evidence needs stronger validation after fine-tuning. Appendix E.1 states, “We fit one lens on the original model from a small task-agnostic set of texts and read every model through it.” (page 23), leaving representation drift as an alternative explanation for reduced falsity readouts. I recommend testing each checkpoint on independently labeled, non-implanted true and false facts and comparing shared versus checkpoint-specific lenses, to verify that falsity remains detectable after training.

Questions for authors

Please see the Weaknesses section.

Reviewer_1PYZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores doing prompt-conditioned synthetic document fine-tuning (SDF), aiming to produce a single model that believes any fact placed in a <facts> tag in the prompt. Experimental results show that the belief rate on held-out facts rises close to log-linearly with the number of training facts.

Critical strengths
  • The paper explores doing prompt-conditioned SDF and shows that the belief rate on held-out facts rises roughly log-linearly with the number of training facts.
Critical weaknesses
  • Section 4.4 claims prompt-conditioned SDF "has fewer side effects than per-fact SDF". However, the untrained Qwen3.6-27B blackmails in 121/240 samples, while the prompt-cond. SDF + RL blackmails in 33/240 samples. For the trained model the rate with no fact in the tag is 13/120, and 11/60 with no goal added. If my understanding is correct, we would not expect the training to change the model this much?
  • As mentioned in section 5.2, the held-out facts come from the same synthetic bank written by the same generator. Hence, the generalization shown by the scaling curve is actually inside the distribution of the generator.
Questions for authors
  • Could you please address the weaknesses above?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.