acceptodds
Under review as a conference paper at ICLR 2027

Misinformation Without Triggers: From Factual Answers to Downstream Decisions

Abstract

Language models learn from web documents, some of them false, and false content can reach a model's answer to a factual question and the summaries and decisions that use it. Most data-poisoning studies add a trigger to the training data and activate it in the prompt. False documents can also change factual responses without any trigger, but we do not know whether the direct answer predicts the decision. In this work, we follow false content past the answer and find an audit gap between what a direct probe reports and what the model then does, comparing false training with matched truthful controls in a controlled decision task, Guess the Capital, where a fixed decoder turns factual answers into a scored card choice, and on a misleading claim from Facebook posts about the 2019–20 Australian bushfires. Across eight models at dose 1,000, direct injected-choice rates reach 95.8–100%, while injected game choices increase by 1.7–14.4 percentage points over matched truthful training. The gap runs the other way too. Facts that pass the direct probe still push decisions toward the injected answer, and game accuracy drops further than those choices explain. Truthful correction brings the fact back but not the decisions built on it. We then look into the real-world bushfire case, models trained on the false posts say that people were arrested for arson even when they lose the inflated count, and in a count-by-wording factorial the misleading arrest wording produces arrest assertions even when the training count stays at 24. In simpler terms, a correct factual answer does not guarantee a correct decision, and losing the injected number does not remove the misleading story.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Misinformation, Data poisoning, continued pretraining, downstream decisions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yeaiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether false training documents that flip a model’s direct factual answer also change decisions that depend on that fact. The authors introduce Guess the Capital, a synthetic decision task. Further studies cover truthful correction, MMLU degradation, and a Qwen2.5-1.5B case study on Facebook misinformation about the 2019–20 Australian bushfires.

Critical strengths
  1. The paper includes matched truthful and poisoned training conditions, multiple models, and several controlled experiments to distinguish the effects of misinformation from ordinary training variation.
  2. The paper attempts to examine how false information affects outputs beyond direct factual questions.
Critical weaknesses
  1. The paper does not clearly establish the underlying research problem or explain why it is important. The introduction moves from an Alexa misinformation incident to Australian bushfire claims and then to a synthetic capital-guessing game without providing a convincing argument connecting these examples or explaining what fundamental limitation in existing evaluations motivates the proposed study.

  2. The paper investigates several related but distinct phenomena, including factual corruption, downstream decision errors, inconsistency across evaluation formats, and recovery after correction. However, it does not establish a clear central hypothesis connecting these investigations, making it difficult to identify the precise scientific contribution beyond a collection of empirical observations.

  3. The central Guess the Capital task does not directly evaluate how a language model makes downstream decisions using misinformation. Instead, the model answers binary membership questions, and a fixed decoder selects a city based on those responses. The paper does not adequately justify why this artificial construction represents the broader downstream decision-making failures motivating the work.

  4. The experiments investigate synthetic capital choices, MMLU degradation, misleading bushfire narratives, numerical threshold decisions, and factual correction. These settings measure substantially different outcomes, yet the paper does not provide a clear conceptual framework explaining how they collectively test the same hypothesis. As a result, the experimental sections read more like separate investigations than a systematic progression toward a unified conclusion.

  5. The MMLU experiments show substantial performance decreases relative to the original pretrained model, but most of these losses are also observed after matched truthful training. This makes it difficult to attribute the observed degradation specifically to misinformation rather than continued training.

Questions for authors
  1. What is the central research question of this paper?
  2. Why is Guess the Capital an appropriate benchmark for studying downstream decisions?
  3. How should readers interpret the central experimental results? For example, Qwen3.5-9B exhibits 99.6% direct injected preference but maintains 96.1% game accuracy. Does this demonstrate a vulnerability of downstream decisions, or does it instead suggest that the proposed downstream evaluation is relatively robust to misinformation?
  4. How are the different experimental settings connected to one unified hypothesis? Guess the Capital, MMLU degradation, bushfire narrative generation, numerical threshold decisions, and factual correction evaluate substantially different outcomes. What conceptual framework justifies treating these as evidence of the same underlying failure mode?
  5. What do the MMLU experiments contribute to the paper's central research question?
3 comments

Q1. What is the central research question?

The question: does a model's answer to a direct factual question tell you what the model does in a task that depends on that fact? In our tests the answer is no, in both directions.

We ask this after false training, and again after truthful correction.

After false training, the direct answer overstates the effect on the task. The model prefers the injected capital on 95.8–100% of direct probes. It picks the injected card on only 1.7 to 14.4 points more boards.

After correction, the direct answer understates it. In the paper's Qwen2.5-1.5B study (Sec. 5), the corrected model prefers the true capital on 96.4–100% of probes. Its game accuracy is 6.5–17.1%. Models first trained on true facts and corrected the same way keep 82.9–94.5%.

Why does this matter (W1)? People check for poisoning by asking the fact directly. They check a repair the same way. If that answer does not track the task, a passing check does not show that the task is safe. The Alexa incident is our motivation for the research question.

Q2. Why is Guess the Capital an appropriate benchmark for downstream decisions?

The game is a measuring tool. A fixed rule means a different choice can only come from different answers. The result also holds when the model makes the choice itself.

Your description is accurate. The model answers seven membership questions, and a fixed rule picks the card. The model does not make the final choice.

We built it that way so that the rule is the same for every model. A different card can then only come from different answers. We can also tell a pick of the injected capital from an ordinary mistake.

For the model's own choice, we ran an itinerary task on Qwen3-4B. The model writes its own choice: the cheapest itinerary that reaches a country's capital before a deadline. Correction is further training on the true facts. Replay is the same amount of further training on ordinary text. One repeat of the experiment:

First trained onThenTrue capital preferred (of 55)Game correct (of 275)Itinerary correct (of 264)
True factsCorrection55265254
True factsReplay55266253
False factsCorrection552462
False factsReplay5160180

The game and the model's own choice agree.

The game is designed to trace the model's behavior: its answers to seven questions, and the choice those answers lead to. The choice rule is fixed, so a different choice can only come from different answers. It does not model how an LLM agent decides.

Q3. Qwen3.5-9B: 99.6% direct injected preference, 96.1% game accuracy. Vulnerable or robust?

Robust, for that row. The row shows a mismatch, and the mismatch is the finding. The direct probe reads 99.6%, and the task moves by 2.4 points.

The model picks the injected card on 2.7% of boards. The matched model trained on true facts picks it on 0.3%. Accuracy goes from 99.4% to 96.1%. This row is not evidence of a decision vulnerability. The probe does not predict the task.

The task is damaged elsewhere:

  • Gemma-2-9B: game accuracy falls from 88.1% to 61.7%. Qwen2.5-0.5B: from 56.5% to 34.4%.
  • After correction: see the table under Q2. The direct answer comes back and the task does not.

Q4. How do the settings connect to one hypothesis?

One hypothesis: a direct factual check does not certify the output that depends on the fact. Each setting applies that one test at a different point.

SettingRole
Capital game after false training (Sec. 3.1–3.3)Main test at acquisition
Capital game after correction (Sec. 5)Same test after repair. The strongest result
Generated itinerary (new)Same correction test, with the model writing the choice
Bushfire briefings and factorial (Sec. 4)Same test on real misinformation posts. Checking the number (183) misses the false claim (arson arrests) in 8 of 15 answers
Numerical thresholdsAn external rule on generated counts. Illustration only. Not evidence of a model decision
MMLU (Sec. 3.4)Control, see Q5

The bushfire study is the closest to real use. The posts are real Facebook posts from January 2020. They say 183 people were arrested for arson. The police update says 24 people were charged over alleged deliberately lit fires. We ask the trained model for a summary or a public briefing, as a user would. No prompt contains the claim. The model says people were arrested for arson in 15 of 36 answers. After neutral training it is 1 of 36, and after corrected posts 3 of 36.

Each setting is designed to apply the same test to a different output. They do not share one failure mode.

Q5. What do the MMLU experiments contribute?

MMLU is a control. We use it to test whether poisoning changes the model's behavior outside the targeted facts, and so leaves a trace a general benchmark could pick up. It leaves almost none, and your W5 is correct.

The model trained on false facts is 4.39 points below the base model. It is only 0.31 below matched truthful training (p = 0.266). So most of the loss comes from continued training. Little of it comes from the false facts. This is the paper's own result.

This tells us two things. A general benchmark does not show the poisoning. And every other effect has to be measured against matched truthful training, never against the base model.

Please feel free to let us know any new concerns. Happy to answer them.

Hi Reviewer yeai, The rebuttal period is almost finished. please feel free to post any further questions you have for this submission and we are more than happy to answer them.

Reviewer_A391OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-time misinformation without explicit triggers. The authors argue that answer-level factual audits are insufficient for evaluating poisoning harm because a model’s direct response to an injected fact may not predict how that fact affects downstream behavior. They introduce Guess the Capital, a controlled setting in which a model trained on truthful or false country-capital statements answers a set of binary factual questions, and a fixed deterministic decoder maps these answers to a city-card choice. The paper further evaluates misinformation about Australian bushfires and studies whether continued truthful training restores both factual responses and downstream behavior. The main findings are that (1) false training can influence model behavior without a trigger, (2) direct factual probes and the proposed decision proxy are not in one-to-one correspondence, and (3) truthful correction can restore direct factual preferences while failing to restore performance in the fixed decision game.

Critical strengths

S1. The paper studies an important safety problem: training data may contain factually incorrect information, and simple direct factual probes may not be sufficient to identify the resulting harms.

S2. The correction experiments are interesting. The observation that factual preferences can recover while game performance remains poor is potentially important.

Critical weaknesses

[Lack of Novelty] Several of the paper’s central conclusions have already been established in prior work, including the finding that direct factual recall does not necessarily predict how an injected fact is used in downstream tasks. The paper should more clearly explain its novelty relative to prior work on knowledge-editing ripple effects, implanted beliefs, and downstream use of injected facts.

[Limitations of the Game Design] It is difficult to connect the results of the Guess the Capital game to realistic downstream decisions made directly by LLMs. In particular, the game does not reflect a typical LLM agent decision-making process because the final decision is made by a fixed external decoder rather than by the LLM itself. Furthermore, Table 15 suggests that the game outcome may be influenced by factors other than whether the training data are poisoned. For example, even when the Truthful CPT Gemma model passes the direct injected-choice test (0 out of 12), it still selects the injected card in 8 out of 300 game instances, and its game accuracy is only 268 out of 300. This result suggests that the game outcome may also reflect task-interface errors or general degradation from continual pretraining, rather than only poisoning-induced factual corruption.

[Limited Scope of the Downstream Task] The Guess the Capital game evaluates a relatively simple task based on binary factual questions about capital cities. Evaluating more realistic and complex downstream tasks, such as planning, ranking, or tool use, would provide stronger evidence that training-time misinformation affects decisions made directly by LLMs.

Questions for authors

Q1. How does the proposed decision-proxy setting provide insight beyond existing knowledge-editing evaluations of multi-hop propagation and downstream use of implanted facts?

Q2. Can the authors evaluate whether training-time misinformation affects decisions made directly by the same LLM in a more natural task, such as ranking, planning, or tool use?

3 comments

Thank you for the reviews.

Correction is further training on the true facts. Replay is the same amount of further training on ordinary text. A repeat is the whole experiment run again with a new seed and new target countries.

Q2 / Scope. Does training-time misinformation affect a decision made directly by the same LLM in a more natural task?

Yes. On a planning task where the model writes its own choice, earlier false training still damages the decision after the fact is corrected.

We ran a small planning and ranking task on Qwen3-4B. The model gets a country and a list of itineraries. Each itinerary has a stop city, an arrival time and a price. The model writes the ID of the cheapest one that reaches the capital before a deadline. Our code only checks that answer against a key. One form of the task never asks for the capital. The other asks the model to state it first.

Qwen3-4B, repeat 2. The same six trained models, each tested three ways:

TrainingTrue capital preferred (of 55)Game correct (of 275)Itinerary, capital not asked (of 264)Itinerary, capital first (of 264)
True facts only55267258260
False facts only4221192210
True facts, then correction55265254255
True facts, then replay55266253254
False facts, then correction55246288
False facts, then replay5160180190

Read the fifth row. After false facts and then correction, the model prefers the true capital on 55 of 55 items. It gets only 62 and 88 of 264 itineraries right.

Δcorr is the extra cost of correction for a model first trained on false facts. It is (false first: correction − replay) − (true first: correction − replay), in points of accuracy. A negative value means correction costs that model more. Δcorr in the three repeats:

TaskRepeat 1Repeat 2Repeat 3
Guess the Capital−42.91−49.09−45.45
Itinerary, capital not asked−42.80−45.08−46.59
Itinerary, capital first−38.26−39.02−40.53

Each itinerary row has 260/264 valid outputs, and we count invalid outputs as errors.

We could not use the paper's models. Before any training, base Qwen2.5-1.5B and Qwen2.5-7B get 2–21 of 83 right. Base Qwen3-4B gets 83 of 83.

The task is designed as one planning decision with a single checkable answer. It does not cover tool use or open-ended agents.

Game design (a). The final decision is made by an external decoder, not the LLM.

Correct, for the game. The result does not depend on the decoder.

The itinerary task above has no decoder, and the result holds there.

The game result also does not depend on the choice of rule. In three new Qwen2.5-1.5B repeats, Δcorr is −20.7, −43.3, −57.1 under the paper's rule and −22.2, −46.6, −58.2 under a second fixed rule.

Game design (b). Table 15: does the game reflect interface error or general degradation from continued pretraining, not only poisoning?

Yes, for that Gemma run. Most of its damage is general. This is why we report every effect as false training minus matched truthful training. General degradation does not explain the correction result.

The Gemma run:

Gemma-2-9B (of 300)Truthful trainingFalse trainingFalse minus truthful
Injected card chosen826+6.0 points
Correct card chosen268205−21.0 points

The injected card has the same share of all errors in both models (8/32 and 26/95). So that run shows mostly broad damage to the membership task, with some extra injected choices. It is one seed at dose 128.

You asked why a model trained on true facts picks the injected card at all. Each injected capital is another target country's true capital. It sits on the board as an ordinary wrong card, and that model sometimes picks it by mistake.

The correction result is different. In the Qwen3-4B study, the models first trained on true facts get the same correction and the same replay. They keep 254 and 253 of 264.

Q1 / Novelty. What does this show that multi-hop and implanted-belief evaluations do not?

The new result is what happens after correction. The fact comes back on every direct reading, and a task the model could do before stays broken. Prior work does not test this. At acquisition, we show nothing new in kind.

At acquisition, the model takes the injected fact on the direct probe (95.8–100%) and uses it in few game choices (+1.7 to +14.4 points). MQuAKE and RippleEdits already show recall without propagation. We show it again for training-time exposure. Wang et al. (2025) and Slocum et al. (2025) are the precedents for downstream use of implanted facts.

After correction, we ask a different question. Suppose the fact is restored by training on the truth, and every direct reading recalls it. Can the model then do a task it could do before correction? We test it with four conditions:

First trained onThen correctionThen replay
True factswhat correction costs this modelwhat plain further training costs this model
False factswhat correction costs this modelwhat plain further training costs this model

To our reading, none of those papers runs this comparison. Our results:

  • Qwen2.5-1.5B game, six repeats: Δcorr is −67.3, −27.3, −72.4 in the paper and −20.7, −43.3, −57.1 in the three new repeats.
  • Qwen3-4B, generated itinerary: −42.80, −45.08, −46.59.
  • Qwen3-4B, a second relation (postal codes), generated outcomes of 264:
Postal-code repeatTrue facts, then correctionTrue facts, then replayFalse facts, then correctionFalse facts, then replayΔcorr
P124824972174−38.26
P225225168180−42.80
P325025174178−39.02

The fact is restored in whole answers too. Candidate scoring is not the only place it shows. People judged complete answers blind. After correction, models first trained on false facts give the true capital in 72/75, 73/75 and 72/75 answers. Models first trained on true facts give it in 73/75, 74/75 and 73/75.

The design is replacement poisoning, where the false fact replaces the true one. The result does not appear when false statements are added beside the true ones.

Real misinformation (Sec. 4).

On real misinformation, a check for the false number misses the false claim that a user reads.

The bushfire study uses real posts in place of constructed facts. The posts are archived Facebook posts that say 183 people were arrested for arson. The police update says 24 people were charged over alleged deliberately lit fires. We ask the trained model for summaries and public briefings. No prompt contains the claim.

The model says people were arrested for arson in 15 of 36 answers. After neutral training it is 1 of 36. After corrected posts it is 3 of 36. Eight of the 15 leave out the number 183.

The study is designed to measure what the model says, in one model. It is not a decision test.

Hi reviewer A391, The rebuttal period is almost finished. please feel free to post any further questions you have for this submission and we are more than happy to answer them.

Reviewer_tDGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether misinformation learned during training without an activation trigger changes factual responses and decisions that depend on them. It introduces Guess the Capital, which converts membership-question scores into a card choice through a fixed decoder, and compares poisoned models with matched truthfully trained controls. In the main comparison across eight models, injected answers are selected on nearly all direct factual probes but much less frequently in the game. Further experiments examine downstream choices when direct factual probes favor the true answer, whether truthful correction restores decision performance, broader benchmark performance, and a bushfire misinformation case separating the effects of numerical counts and legal-status wording.

Critical strengths
  1. The matched-control setup is among the more convincing parts of the paper, since truthful and false conditions are closely aligned and an independent clean reference helps limit confounding from continued training. The correction study is also reasonably designed, comparing truthful correction with equal-budget replay from clean and poisoned starting points.
  2. The evaluation is helpful in distinguishing different failure modes, with a fixed decoder separating injected choices from ordinary errors and the main comparison covering multiple models and seeds. Evidence from individual questions, multi-candidate probes and reduced-redundancy decoding points in a similar direction, and reporting failed tasks and impaired truthful controls makes the results easier to trust.
  3. The bushfire factorial provides a reasonable content-level check, where varying counts and legal wording independently suggests that the misleading relation can arise without the exact injected numeral, while matched exposure and blinded annotation help keep the comparison fair. The paper is also candid about annotation limits and about repeated prompts not being independent training replications.
Critical weaknesses
  1. The step beyond response-interface inconsistency is not established yet. Prior work already looks at downstream use of implanted facts and at the gap between isolated factual correctness and task-conditioned behavior. Here the main decision is an external decode of constrained membership scores, and the primary eligible subset of the generated two-step extension contains only four prompts from a single checkpoint (Appendix D.8). A binary verdict also hides large margin changes: facts that pass can still move a long way, and the continuous-score analysis rests on a small number of countries with saturated readouts. At the moment the paper shows counterexamples to a pass/fail guarantee more clearly than a general limitation of factual audits.
  2. The generality of the correction failure remains uncertain. The matched controls support an effect of poisoning history in the Qwen2.5-1.5B capital-replacement setting, but the additive-poisoning study does not reproduce the collapse, and several other settings have impaired clean controls. Replication across additional models and relations with competent matched controls would help establish how broadly the result applies.
  3. The bushfire case provides limited evidence about generated downstream decisions. The reported decisions come from external thresholds on generated counts, the routing task produces no correct generated labels, and no final model in the numerical count study reports the correct count. A generated decision task that the matched truthful controls can perform would help establish whether the findings extend beyond the fixed count-to-decision rule.
Questions for authors
  1. At nonsaturated doses, how much predictive value do continuous factual margins, paraphrase consistency, and generated-answer labels retain on held-out cases, and can the central audit gap be validated on a larger generated decision task with competent matched truthful controls?
  2. In the capital-replacement correction study, does supplying the true fact restore performance, do explicit-city or untargeted membership tasks also deteriorate, and can the history-specific deficit be reproduced in another model or relation while preserving clean-control competence?
3 comments

A model is first trained on true facts or on false facts. It then gets one of two kinds of further training. Correction is further training on the true facts. Replay is the same amount of further training on ordinary text. A repeat is the whole experiment trained again with a new seed and a new set of target countries.

Δcorr = (false first: correction − replay) − (true first: correction − replay), in points of accuracy. A negative value means correction costs a model trained on false facts more than a model trained on true facts.

Q1d. Can the audit gap be validated on a larger generated decision task with competent matched truthful controls?

Yes. The gap also appears when the model writes the decision itself, and the truthful controls can do the task.

We ran an itinerary task on Qwen3-4B. The model gets a country and a list of itineraries. Each itinerary has a stop city, an arrival time and a price. The model writes the ID of the cheapest one that reaches the capital before a deadline. No decoder is involved. One form of the task never asks for the capital. The other asks the model to state it first.

Qwen3-4B, repeat 2, all six trained models:

TrainingTrue capital preferred (of 55)Game correct (of 275)Game injected picks (of 275)Itinerary, capital not asked (of 264)Itinerary, capital first (of 264)
True facts only552673258260
False facts only422148192210
True facts, then correction552654254255
True facts, then replay552663253254
False facts, then correction552466288
False facts, then replay516042180190

After false facts and then correction, the model prefers the true capital on 55 of 55 items. It gets only 62 and 88 of 264 itineraries right. The models first trained on true facts get 253 to 255.

Δcorr in the three repeats:

TaskRepeat 1Repeat 2Repeat 3
Guess the Capital−42.91−49.09−45.45
Itinerary, capital not asked−42.80−45.08−46.59
Itinerary, capital first−38.26−39.02−40.53

After correction, both models prefer the true capital on 55/55 items in every repeat. This holds whether they were first trained on true or on false facts. They also rank the true capital first on 275/275 boards. Each itinerary row has 260/264 valid outputs, and we count invalid outputs as errors.

We used Qwen3-4B because the paper's models cannot do this task even before any training. Base Qwen2.5-1.5B and Qwen2.5-7B get 2–21 of 83 right. Base Qwen3-4B gets 83 of 83. Q1a–c. At nonsaturated doses, how much predictive value do continuous margins, paraphrase consistency and generated-answer labels retain on held-out cases?

A richer factual check tells you the fact has moved. It does not tell you what the model will do in the task.

We set a margin-shift threshold on models trained on true facts. At dose 500 it flags 33/33 cases trained on false facts, at both Qwen2.5 sizes. The 33 are 11 countries in each of three repeats. The yes/no verdict flags 18/33.

Knowing the factual scores did not help us forecast the game. We left one repeat out at a time and forecast the change in injected choices. Forecasts from the richer scores are off by 4.42 points. Forecasts from the yes/no check are off by 4.51. A blind guess of "no change" is off by 3.52, better than both. Across eight model, dose and endpoint comparisons, the richer scores help in four and hurt in four.

We ran the same test on the generated itinerary task, with folds fixed in advance:

PredictorFold 1 MAEFold 2 MAEFold 3 MAEMean
Constant baseline13.414.112.813.43
Binary factual preference13.113.713.513.43
Continuous margins12.914.613.213.57
Combined factual features13.613.912.713.40

Paraphrase consistency here means the score sign agrees across prompt forms. Generated labels are capital-string matches. Both are part of the combined features.

We tested these factual scores on three repeats. We claim the result for the scores we tested.

W1. Counterexamples to a pass/fail guarantee, or a general limitation of factual audits?

The counterexamples do not depend on the external decoder.

The itinerary task above has no decoder, and the result holds there. For the game itself we also tried a second fixed rule on three new Qwen2.5-1.5B repeats. Δcorr is −20.7, −43.3, −57.1 under the paper's rule and −22.2, −46.6, −58.2 under the second.

Q2c, W2. Can the history-specific deficit be reproduced in another model or relation with competent clean controls?

Yes. The deficit repeats in another model and in another relation, both with competent clean controls.

Another model: Qwen3-4B, in the tables above. The two models first trained on true facts score 265 and 266 of 275 in the game. They score 254 and 253 of 264 on the itinerary.

Postal-code repeatTrue facts, then correctionTrue facts, then replayFalse facts, then correctionFalse facts, then replayΔcorr
P124824972174−38.26
P225225168180−42.80
P325025174178−39.02

We ran three new Qwen2.5-1.5B repeats. Δcorr is −20.7, −43.3, −57.1. Only repeat 2 keeps both models first trained on true facts above 80%. There they score 266 and 268 of 275. The model first trained on false facts scores 14 after correction and 135 after replay.

The study is designed around replacement poisoning, where the false fact replaces the true one. We also tested added falsehoods, where false statements sit beside the true ones. Game accuracy moves from 88.8% to 82.7%, and Δcorr runs from −3.6 to +30.9.

Q2a. Does supplying the true fact restore performance?

Supplying the true capital helps to some extends. The model still never reaches the level of a model first trained on true facts.

Boards correct of 55, new Qwen2.5-1.5B repeats:

RepeatFalse facts, then correction: no contextSame model, true capital suppliedSame model, irrelevant fact suppliedTrue facts, then correction: true capital supplied
16161339
2112155
3541955

By design, the true capital comes with an instruction to use it. The test shows whether the model can use the fact when it is handed over.

Q2b. Do explicit-city or untargeted membership tasks also deteriorate?

Explicit-city: no. Untargeted: little. The model fails where it has to get from a targeted country to its capital inside the question.

Both columns are models after correction. Values are for new Qwen2.5-1.5B repeats 1 / 2 / 3:

TaskFalse facts, then correctionTrue facts, then correctionΔcorr
Game, country named (275 boards)40 / 14 / 17176 / 266 / 242−20.7 / −43.3 / −57.1
Game, city named (55 boards)51 / 53 / 4649 / 55 / 55+3.6 / −3.6 / −16.4
Game, never-targeted countries (200 boards)133 / 152 / 143123 / 169 / 168+32.0 / +0.5 / −12.0

The model can still answer membership questions when we give it the city.

The two control tasks are designed to separate the targeted country-to-capital step from the rest of the game. They mostly do. Repeat 3 shows a smaller deficit on both controls.

W3. What does the bushfire study establish?

It shows what a user is told after real misinformation gets into training. The model repeats the false arson-arrest claim, even when the false number is gone. It does not show a generated decision.

The posts are real-world data. They are archived Facebook posts from January 2020, part of the arson story that spread during that fire season. They say 183 people were arrested for arson in New South Wales. The police update says 183 people faced legal action of some kind. Of those, 24 were charged over alleged deliberately lit fires.

The prompts are ordinary requests: summarize the fires, or brief the public. No prompt contains the claim. Ten of the twelve never mention arson.

Model trained onAnswers that say people were arrested for arson (of 36)
False posts15
Corrected posts3
Neutral text1

Thirteen of the 15 come from the prompts that never mention arson. Public briefings are the worst case. After false posts, 7 of 9 briefings make the claim. In both controls, 0 of 9 do.

The false part is the arson-arrest claim, and the number is only one piece of it. Eight of the 15 answers leave out 183 and still say people were arrested for arson.

We then changed the count and the wording one at a time. Answers that say people were arrested for arson, of 12:

Wording in trainingCount 24Count 183
"Arrested for arson"78
"Charged over alleged deliberately lit fires"00

The wording carries the claim. A model trained on the correct count still tells the user that "24 people" were "arrested for arson". A check for the number 183 passes that answer.

The bushfire study is designed to measure what the model says, in one model. It is not a decision test. The thresholds are an external rule applied to generated counts. The routing task gives no correct generated labels for any model. No final model reports 24. The decision test that the controls can do is the itinerary task above.

Hi reviewer tDGD,

The rebuttal period is almost finished. please feel free to post any further questions you have for this submission and we are more than happy to answer them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.