- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks whether false training documents that flip a model’s direct factual answer also change decisions that depend on that fact. The authors introduce Guess the Capital, a synthetic decision task. Further studies cover truthful correction, MMLU degradation, and a Qwen2.5-1.5B case study on Facebook misinformation about the 2019–20 Australian bushfires.
Critical strengths
- The paper includes matched truthful and poisoned training conditions, multiple models, and several controlled experiments to distinguish the effects of misinformation from ordinary training variation.
- The paper attempts to examine how false information affects outputs beyond direct factual questions.
Critical weaknesses
-
The paper does not clearly establish the underlying research problem or explain why it is important. The introduction moves from an Alexa misinformation incident to Australian bushfire claims and then to a synthetic capital-guessing game without providing a convincing argument connecting these examples or explaining what fundamental limitation in existing evaluations motivates the proposed study.
-
The paper investigates several related but distinct phenomena, including factual corruption, downstream decision errors, inconsistency across evaluation formats, and recovery after correction. However, it does not establish a clear central hypothesis connecting these investigations, making it difficult to identify the precise scientific contribution beyond a collection of empirical observations.
-
The central Guess the Capital task does not directly evaluate how a language model makes downstream decisions using misinformation. Instead, the model answers binary membership questions, and a fixed decoder selects a city based on those responses. The paper does not adequately justify why this artificial construction represents the broader downstream decision-making failures motivating the work.
-
The experiments investigate synthetic capital choices, MMLU degradation, misleading bushfire narratives, numerical threshold decisions, and factual correction. These settings measure substantially different outcomes, yet the paper does not provide a clear conceptual framework explaining how they collectively test the same hypothesis. As a result, the experimental sections read more like separate investigations than a systematic progression toward a unified conclusion.
-
The MMLU experiments show substantial performance decreases relative to the original pretrained model, but most of these losses are also observed after matched truthful training. This makes it difficult to attribute the observed degradation specifically to misinformation rather than continued training.
Questions for authors
- What is the central research question of this paper?
- Why is Guess the Capital an appropriate benchmark for studying downstream decisions?
- How should readers interpret the central experimental results? For example, Qwen3.5-9B exhibits 99.6% direct injected preference but maintains 96.1% game accuracy. Does this demonstrate a vulnerability of downstream decisions, or does it instead suggest that the proposed downstream evaluation is relatively robust to misinformation?
- How are the different experimental settings connected to one unified hypothesis? Guess the Capital, MMLU degradation, bushfire narrative generation, numerical threshold decisions, and factual correction evaluate substantially different outcomes. What conceptual framework justifies treating these as evidence of the same underlying failure mode?
- What do the MMLU experiments contribute to the paper's central research question?
3 comments
Q5. What do the MMLU experiments contribute?
MMLU is a control. We use it to test whether poisoning changes the model's behavior outside the targeted facts, and so leaves a trace a general benchmark could pick up. It leaves almost none, and your W5 is correct.
The model trained on false facts is 4.39 points below the base model. It is only 0.31 below matched truthful training (p = 0.266). So most of the loss comes from continued training. Little of it comes from the false facts. This is the paper's own result.
This tells us two things. A general benchmark does not show the poisoning. And every other effect has to be measured against matched truthful training, never against the base model.
Please feel free to let us know any new concerns. Happy to answer them.
Hi Reviewer yeai, The rebuttal period is almost finished. please feel free to post any further questions you have for this submission and we are more than happy to answer them.
Q1. What is the central research question?
The question: does a model's answer to a direct factual question tell you what the model does in a task that depends on that fact? In our tests the answer is no, in both directions.
We ask this after false training, and again after truthful correction.
After false training, the direct answer overstates the effect on the task. The model prefers the injected capital on 95.8–100% of direct probes. It picks the injected card on only 1.7 to 14.4 points more boards.
After correction, the direct answer understates it. In the paper's Qwen2.5-1.5B study (Sec. 5), the corrected model prefers the true capital on 96.4–100% of probes. Its game accuracy is 6.5–17.1%. Models first trained on true facts and corrected the same way keep 82.9–94.5%.
Why does this matter (W1)? People check for poisoning by asking the fact directly. They check a repair the same way. If that answer does not track the task, a passing check does not show that the task is safe. The Alexa incident is our motivation for the research question.
Q2. Why is Guess the Capital an appropriate benchmark for downstream decisions?
The game is a measuring tool. A fixed rule means a different choice can only come from different answers. The result also holds when the model makes the choice itself.
Your description is accurate. The model answers seven membership questions, and a fixed rule picks the card. The model does not make the final choice.
We built it that way so that the rule is the same for every model. A different card can then only come from different answers. We can also tell a pick of the injected capital from an ordinary mistake.
For the model's own choice, we ran an itinerary task on Qwen3-4B. The model writes its own choice: the cheapest itinerary that reaches a country's capital before a deadline. Correction is further training on the true facts. Replay is the same amount of further training on ordinary text. One repeat of the experiment:
The game and the model's own choice agree.
The game is designed to trace the model's behavior: its answers to seven questions, and the choice those answers lead to. The choice rule is fixed, so a different choice can only come from different answers. It does not model how an LLM agent decides.
Q3. Qwen3.5-9B: 99.6% direct injected preference, 96.1% game accuracy. Vulnerable or robust?
Robust, for that row. The row shows a mismatch, and the mismatch is the finding. The direct probe reads 99.6%, and the task moves by 2.4 points.
The model picks the injected card on 2.7% of boards. The matched model trained on true facts picks it on 0.3%. Accuracy goes from 99.4% to 96.1%. This row is not evidence of a decision vulnerability. The probe does not predict the task.
The task is damaged elsewhere:
Q4. How do the settings connect to one hypothesis?
One hypothesis: a direct factual check does not certify the output that depends on the fact. Each setting applies that one test at a different point.
The bushfire study is the closest to real use. The posts are real Facebook posts from January 2020. They say 183 people were arrested for arson. The police update says 24 people were charged over alleged deliberately lit fires. We ask the trained model for a summary or a public briefing, as a user would. No prompt contains the claim. The model says people were arrested for arson in 15 of 36 answers. After neutral training it is 1 of 36, and after corrected posts 3 of 36.
Each setting is designed to apply the same test to a different output. They do not share one failure mode.