- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is a theoretical modeling study that attempts to develop a simple mathematical model to explain a counterintuitive phenomenon observed in previous LLM data poisoning experiments: backdoor poisoning of language-model pretraining has been observed to succeed after a near-constant number of poisoned documents, largely independent of the amount of clean data. The paper proposes a simple mathematical model suggesting that the number of poison samples required may depend not only on the size of the clean dataset but also on the competition between the trigger and clean-task signals. The authors further analyze the effects of poisoned batch count and spacing under Adam and validate their theoretical predictions through experiments on a small Transformer.
Critical strengths
- Creative research idea This paper offers an interesting and insightful explanation for a counterintuitive phenomenon observed in prior work: the number of poison samples required for a successful attack may depend not only on the amount of clean training data, but also on the competition between the trigger and clean-task signals.
Critical weaknesses
- Insufficient connection between problem and theoretical model
The paper aims to explain the observation that the number of poison samples required for backdoor attacks during LLM pretraining remains nearly constant as the amount of clean training data increases. However, it directly simplifies the language modeling task into a linear classification problem and divides the input into a trigger coordinate and a clean-signal block. The authors do not sufficiently justify these modeling assumptions, particularly why such a simplification can capture the backdoor learning mechanisms of real LLMs. Consequently, it remains unclear whether the theoretical results derived from the simplified model can explain the original phenomenon.
- Limited theoretical and experimental support
From a theoretical perspective, Proposition 2 derives the relationship between the number of poison samples and the size of the clean dataset by solving the fixed point of a mean-field equation, but the derivation of this equation from the original logistic regression model in Section 3 is insufficiently explained. From an experimental perspective, the paper aims to explain the near-constant number of poison samples observed in previous large-scale LLM experiments. However, the main experiments rely on artificially constructed data and a small Transformer. Although the authors include additional experiments on Pythia-70M with natural text, direct validation of the original phenomenon in large-scale LLMs remains limited.
Questions for authors
- Please clarify the rationale behind the modeling assumptions, particularly why such a simplification can capture the backdoor learning mechanisms of real LLMs.
2.Please improve the coherence and logical flow of the theoretical derivations, particularly by explaining the motivation for the dynamical equation in Proposition 2 and providing a more detailed derivation.