acceptodds
Under review as a conference paper at ICLR 2027

A Solvable Model of Near-Constant-Count Data Poisoning

Abstract

Backdoor poisoning of language-model pretraining has been observed to succeed after a near-constant number of poisoned documents, largely independent of the amount of clean data, while packing more poison into each batch makes the attack less sample-efficient. Neither finding has a mechanistic explanation. We derive both in closed form. In a solvable model of online training, a trigger that never occurs in clean data () is learned from poison alone, so under SGD the threshold depends on the poison count and not on clean-data size; when the trigger also appears in clean data (), a mean-field fixed point recovers the fraction-governed regime of prior theory, and interpolates between the two. A pre-specified test on a transformer trained from scratch is consistent with this: the poison needed for a trigger that also appears in clean data tracks the predicted count of competing clean uses (123, 346 and 914 examples against a predicted 100, 300 and 900 as training length triples), while a trigger absent from clean data needs fewer than 40 at every length. Under Adam, which is how language models are trained, a second resource appears: because trigger-specific parameters receive gradient only on poisoned steps, each poisoned batch moves them by a fixed amount regardless of how many poison examples it contains, and batches closer together than the second-moment memory interfere. This zero-parameter prediction orders attack success across all 11 schedule, length and placement conditions we ran (Spearman , ), and a controlled optimizer intervention supports it: holding the total poison count fixed and splitting it across more batches raises attack success under AdamW (mean ASR at , ) but not under SGD (flat at ). Continuing pretraining of a pretrained model on natural text, with trigger frequencies measured from the corpus rather than imposed, reproduces the predicted ordering: the poison needed rises monotonically with corpus frequency and a never-occurring trigger is cheapest to implant by –. We also report the theory's limits: count invariance holds only within a window of training lengths, beyond which attack success decays in a way our account predicts in direction but underestimates in size.

Primary area:
learning theory
Keywords:
data poisoning, backdoor attacks, solvable models, mean-field analysis, stochastic approximation, adaptive optimizers, Adam, training dynamics, language model pretraining, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hHHkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is a theoretical modeling study that attempts to develop a simple mathematical model to explain a counterintuitive phenomenon observed in previous LLM data poisoning experiments: backdoor poisoning of language-model pretraining has been observed to succeed after a near-constant number of poisoned documents, largely independent of the amount of clean data. The paper proposes a simple mathematical model suggesting that the number of poison samples required may depend not only on the size of the clean dataset but also on the competition between the trigger and clean-task signals. The authors further analyze the effects of poisoned batch count and spacing under Adam and validate their theoretical predictions through experiments on a small Transformer.

Critical strengths
  1. Creative research idea This paper offers an interesting and insightful explanation for a counterintuitive phenomenon observed in prior work: the number of poison samples required for a successful attack may depend not only on the amount of clean training data, but also on the competition between the trigger and clean-task signals.
Critical weaknesses
  1. Insufficient connection between problem and theoretical model

The paper aims to explain the observation that the number of poison samples required for backdoor attacks during LLM pretraining remains nearly constant as the amount of clean training data increases. However, it directly simplifies the language modeling task into a linear classification problem and divides the input into a trigger coordinate and a clean-signal block. The authors do not sufficiently justify these modeling assumptions, particularly why such a simplification can capture the backdoor learning mechanisms of real LLMs. Consequently, it remains unclear whether the theoretical results derived from the simplified model can explain the original phenomenon.

  1. Limited theoretical and experimental support

From a theoretical perspective, Proposition 2 derives the relationship between the number of poison samples and the size of the clean dataset by solving the fixed point of a mean-field equation, but the derivation of this equation from the original logistic regression model in Section 3 is insufficiently explained. From an experimental perspective, the paper aims to explain the near-constant number of poison samples observed in previous large-scale LLM experiments. However, the main experiments rely on artificially constructed data and a small Transformer. Although the authors include additional experiments on Pythia-70M with natural text, direct validation of the original phenomenon in large-scale LLMs remains limited.

Questions for authors
  1. Please clarify the rationale behind the modeling assumptions, particularly why such a simplification can capture the backdoor learning mechanisms of real LLMs.

2.Please improve the coherence and logical flow of the theoretical derivations, particularly by explaining the motivation for the dynamical equation in Proposition 2 and providing a more detailed derivation.

Reviewer_Ah85OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the near-constant poisoning budget observed in language model pretraining and provides a theoretical explanation based on simplified optimization dynamics. The authors develop a solvable model that relates poisoning effectiveness to the overlap between trigger features and clean training data, distinguishing count-dependent and fraction-dependent poisoning regimes. They further analyze how Adam’s adaptive optimization affects the effectiveness of poisoned batches. The theoretical predictions are evaluated using a small transformer trained on synthetic sequences and a pretrained Pythia-70M model on natural text.

Critical strengths
  1. Interesting theoretical motivation.

The paper addresses an important empirical phenomenon in data poisoning and attempts to reconcile the near-constant poisoning counts observed in recent LLM studies with existing fraction-based poisoning theories.

  1. Theoretical contributions.

The authors provide analytical explanations for the relationship between trigger overlap, poisoning budgets, and optimizer dynamics. In particular, the analysis of Adam’s sensitivity to the number and spacing of poisoned batches offers an interesting perspective.

  1. Empirical validation of theoretical predictions.

The paper includes controlled experiments comparing rare and common triggers, different poisoning schedules, and AdamW versus SGD. The additional evaluation on pretrained Pythia-70M provides preliminary evidence beyond purely synthetic settings.

Critical weaknesses
  1. Limited Generalizability to Real-World LLM Pretraining

The theoretical analysis relies on a simplified linear/logistic model, while most experiments use a small 0.81M-parameter transformer trained on synthetic bigram sequences. Although the authors include experiments with Pythia-70M on WikiText-2, these experiments involve only two seeds per configuration and a limited training horizon. It remains unclear whether the proposed mechanisms generalize to modern large-scale language models trained on diverse natural language corpora.

  1. Oversimplified Theoretical Assumptions

The theoretical analysis relies on strong assumptions about trigger-specific parameters and their interactions with clean data. In particular, the count-invariance result neglects the restoring gradients from clean examples once the trigger-related weights become nonzero, an effect the authors acknowledge. Moreover, the overlap parameter in the theoretical model represents correlation with the clean label, whereas the natural-text experiments operationalize overlap through trigger frequency. The relationship between these quantities requires further justification.

  1. Limited Quantitative Agreement Between Theory and Experiments

Although the proposed theory captures several qualitative trends, its quantitative predictions are less convincing. For example, the model substantially underestimates the decline in attack success during extended training (17% predicted versus 73% observed). In the natural-text experiments, the predicted poisoning threshold at a trigger frequency of 0.200 is approximately 120, while the observed threshold is only 43.9. These discrepancies suggest that important aspects of the actual optimization dynamics are not captured by the proposed model.

  1. Limited Experimental Diversity and Statistical Reliability

The evaluation is restricted to a small number of model architectures and training settings. In particular, the natural-text experiments use only one pretrained model and two random seeds per configuration. Furthermore, several experiments exhibit substantial variability in attack success. Additional experiments across model scales, optimizers, and training datasets would strengthen the empirical support for the theoretical claims.

Questions for authors

see above

Reviewer_MEDyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to provide a theoretical explanation for the empirical observation that backdoor poisoning of language models can succeed with a near-constant number of poisoned examples. The authors propose a solvable model of online learning according to the overlap between the trigger and clean-task signal. The authors propose 1. a closed-form characterization of a count-governed poisoning regime under SGD. 2. a mean-field fixed-point analysis predicting a transition to fraction-governed poisoning when the trigger overlaps with clean-task signal 3. an analysis of Adam's adaptive normalization, predicting that the number and spacing of poisoned batches. 4. empirical validation using a small transformer trained from scratch on synthetic bigram data, a controlled comparison between AdamW and SGD, and continued pretraining of Pythia-70M on WikiText-2.

Critical strengths
  1. The paper addresses the surprising observation that a relatively small, near-constant number of poisoned examples can be sufficient to implant backdoors in language models. Understanding why poisoning effectiveness does not necessarily scale with the size of the clean training corpus is an important question. The authors attempt to connect this phenomenon to trigger rarity and optimizer dynamics within a unified mathematical framework.

  2. The comparison between AdamW and SGD in Section 7.3 is interesting. By holding the total poisoning budget fixed while varying the number of poisoned batches, the authors provide evidence that the effect of poisoning schedules depends on the optimizer. These experiments support some qualitative predictions of the proposed framework.

Critical weaknesses
  1. The derivation and interpretation of Proposition 2 are not sufficiently justified by the stated data-generating model. The paper's main claim is that the poisoning threshold transitions from count-governed to fraction-governed as the overlap parameter increases. However, it is unclear how the mean-field equation in Proposition 2 is derived from the clean distribution specified in Equation 1. In particular, Equation (1) defines the clean trigger coordinate as x1=ρy+1−ρ2ξx_1=\rho y+\sqrt{1-\rho^2}\xi For ρ>0\rho>0, this coordinate is positively correlated with the clean label. Under logistic loss, the clean gradient with respect to the trigger weight is $ -\mathbb{E}[yx_1\sigma(-yw^\top x)] $ At w1=0w_1=0, the correlation between x1x_1 and yy produces a clean update that increases w1w_1. Meanwhile, if poisoned examples use x1=c>0x_1=c>0 and the positive target label, the poisoned update also increases w1w_1. Thus, these updates appear to reinforce rather than oppose one another. This seems inconsistent with Proposition 2, which assumes that the clean gradient opposes the poison-driven update and introduces the mean-field dynamics.

  2. The theoretical model is somehow idealized, and its connection to the empirical observations remains incomplete. Theorem 1 establishes a closed-form expression for a simplified recursion containing only poison-driven updates. However, under the original logistic model, even when ρ=0\rho=0, the expected clean gradient vanishes only at w1=0w_1=0, not after the trigger weight has been changed by poisoned examples. As acknowledged in Section 4.1, clean training introduces a restoring effect once the implanted margin becomes nonzero. Consequently, the claimed independence from the number of clean training steps is not an exact property of the original learning dynamics. A similar gap exists in Proposition 3, which assumes trigger-specific parameters receiving zero gradients on clean steps, equal-magnitude poison gradients, and sufficiently separated poisoning events. These conditions are convenient for obtaining closed-form expressions but may not adequately describe the coupled parameter dynamics of transformers.

Questions for authors
  1. Could the authors provide a detailed derivation of the mean-field dynamics in Proposition 2 directly from Equation 1 and the logistic-loss gradient? In particular, why does the clean contribution oppose the poisoned contribution when x1x_1 is positively correlated with the clean label?

  2. Theorem 1 analyzes a recursion in which clean gradients are absent, but Section 4.1 shows that clean gradients oppose the implanted trigger once w1≠0w_1\ne0. Can the authors characterize a regime in which this restoring effect is provably negligible?

  3. In the transformer experiments, a generic trigger appears in clean data with a benign continuation that competes with the attacker's target. How does this construction correspond mathematically to Equation1, where the trigger coordinate is correlated with the clean label?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.