acceptodds
Under review as a conference paper at ICLR 2027

Why Backdooring Neural Networks is so Easy?

Abstract

Securing modern AI systems against backdoor attacks remains an open challenge and requires fundamentally principled estimates of the adversary’s budget—the poison fraction and trigger strength needed to construct successful yet stealthy attacks. Motivated by recent empirical evidence that poisoning large language models can require a nearly constant number of malicious samples even as clean datasets grow, we derive an exact closed-form analysis of a quadratic neuron trained on a poisoned Gaussian mixture. We show, perhaps counterintuitively, that the same feature-learning dynamics that make neural networks powerful can also make them more vulnerable to backdoors. Specifically, with clean accuracy preserved to first order, , we demonstrate that lazy learning imposes the inverse-square-root scaling for a successful attack, while feature learning induces a quadratic detector whose loss margin scales as , improving the attack budget to . Consequently, nonlinear feature learning substantially reduces the trigger strength required at small poison fractions, thereby in a sense making feature learners more backdoor vulnerable. These results provide a theoretical mechanism consistent with large-scale empirical observations and demonstrate that security audits based on linear heuristics can systematically underestimate backdoor vulnerability in the widely adopted feature-learning regimes.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Backdoor Attacks, Data Poisoning, Feature Learning, Deep Learning Theory, Adversarial Robustness, Scaling Laws, Gaussian Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8PbHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work develops a theoretical analysis of backdoor attacks using a quadratic neuron trained on a poisoned Gaussian mixture. The authors examine how poisoning and trigger strength affect attack success and clean accuracy, comparing a linear regime with a feature-learning regime. They also conduct experiments on synthetic data, use fitted statistical distributions based on image-dataset statistics, and evaluate neural networks under different training regimes. The results suggest that feature learning can reduce the trigger strength required for a successful attack under the assumptions considered. How broadly this finding applies beyond the analyzed setting remains an open question.

Critical strengths
  1. The paper provides an analytical treatment of the relationship between poisoning and trigger strength in a tractable model. Comparing different learning regimes offers a concrete way to understand how the learning process can affect the cost of a backdoor attack. This analytical result is the main strength of the work.
  2. The paper also distinguishes between the trigger strength required to achieve a target attack success rate and the maximum success rate attainable under the model. This helps clarify what the theoretical analysis establishes and makes the conclusions more precise.
Critical weaknesses
  1. The analysis relies on a relatively specific modeling setup, which makes the theoretical results tractable but also raises questions about their applicability to more general settings. It would be helpful to clarify how strongly the conclusions depend on the quadratic model and the assumptions about the feature direction. Without a clearer understanding of these dependencies, the broader implications remain somewhat uncertain.
  2. Although the supplementary optimization experiments provide additional support for the proposed mechanism, the validation is limited to a particular setting. It remains unclear whether similar behavior would be observed under different data distributions or training conditions. More evidence across varied settings, or a clearer discussion of the relevant limitations, would make this part of the paper more convincing.
  3. Some experiments on image datasets use samples drawn from statistical distributions fitted to dataset statistics. Although this provides a controlled way to evaluate the theoretical predictions, it remains unclear whether the same behavior would be observed on the original datasets. The neural-network experiments provide additional evidence, but the link between their results and the proposed theoretical mechanism could be explained more clearly.
Questions for authors
  1. How sensitive are the main conclusions to the assumptions about the feature direction and the data distribution? In particular, which assumptions are essential to obtaining the reported behavior?
  2. How should the results from experiments using fitted statistical distributions be interpreted relative to the neural-network experiments? Could the authors clarify which experiments provide direct evidence for the proposed mechanism?
  3. Under what conditions do the authors expect the main conclusions to extend beyond the current model? A clearer discussion of the limitations of this extension would help readers assess the broader significance of the findings.
Reviewer_yEZVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates the backdoor threat in a victim model through a quadratic neuron trained by least squares on a Gaussian mixture. Within both the linear and the feature-learning training regimes, this work derives that the trigger-aligned quadratic feature, with a given poisoned fraction π\pi, changes the requires trigger-strength α\alpha scaling from α∝π−12\alpha \propto \pi^{\frac{-1}{2}} to α∝π−14\alpha \propto \pi^{\frac{-1}{4}}. Moreover, the analysis also differencitates clean accuracy and attack success rate, showing an asymptotic preservation of clean accuracy under specified scaling limits.

Critical strengths
  1. This work proposes a useful quantitative perspective.

Studying the trade-off between the fraction of poisoned samples and the trigger strength provides a more informative representation of the backdoor's vulnerability than merely reporting attack success at a few selected budgets.

  1. This work provides an interpretable theoretical mechanism.

The distinct differences between linear and quadratic trigger responses offers a concrete hypothesis for understanding how nonlinear representations can make backdoors implanted into a victim model.

Critical weaknesses
  1. The connection between training dynamics and the trigger alignment is not sufficiently established.

The main feature-learning analysis fixes the feature direction to the whitened trigger. Also, Appendix C.4 provides numerical support for learning an aligned direction in a particular setting.

This work demonstrates the alignment of feature direction with the backdoor trigger and claims that this feature make the backdoor implanted. However, the key question "why and how the backdoor is learned during training" is not interpreted

  1. It is unclear whether the theory holds on actual image datasets.

Experiments in Section 4 use the training and evaluation samples that are generated from Gaussian Mixtures which fits to the statistic of image datasets. However, this setting cannot demonstrate the applicability of the proposed theory to the real image datasets. Thus, the authors should provide the direct evidence by conducting experiments on the original image datasets.

  1. The connection to constant-count LLM poisoning is not demonstrated by the scaling law.

The authors refers to the victim scenario of large-scale language models, while the adaptability of proposed theory to the large-scale models such as LLMs is not studied. This work indicates a reduced trigger-strength requirement, but it does not explain why the poisoning backdoor remains with a fixed number of poisoned samples as the dataset grows.

Questions for authors
  1. How does the proposed theory transfer to the classification with multiple classes? The binary classification problem seems insufficient, as the backdoor's behavior is likely associated to different classes, finally making the backdoor a single latent class rather bounded with the target class.

  2. Given a fixed number of poisoned samples, is there probably a trigger-strength threshold, above which the attack success rate remains reasonably high as the amount of clean samples in the training dataset keep increasing?

  3. How would the theoretical analysis transfer to a multi-class classification problem, particularly when the same trigger acts to facilitate an "All-to-One" or an "All-to-All" backdoor? Would the scaling exponent depend on the geometry of the source and target classes?

Reviewer_bzqEOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Prior work studies poisoning for linear, lazy learners, this work extends this to the feature-learning regime by studying the quadratic neuron. When the quadratic term is zero, the model is in the prior linear, lazy setting. In a nutshell, the authors demonstrate that non-linear feature learning is easier to backdoor than in the linear regime. In hindsight, this makes sense since the planted signal is more detectable.

The data model the paper works in is the following. We begin with an initial "clean" distribution with 1/21/2 the mass on inputs in Rd\mathbb{R}^d with positive labels +1+1, and 1/21/2 the mass on negative samples. The poisoned training distribution changes a π/2\pi/2 fraction of the positive labeled points to negative labels −1-1 and additively shifting those points by additive αv\alpha v for some scalar α\alpha and some vector vv. π\pi is called the poison fraction and α\alpha is the trigger strength. The key object of interest is the smallest trigger strength for a given attack success rate a∈(0,1)a \in (0, 1) and a poison fraction π\pi, where the attack success rate is the probability that a noisy poisoned example is labeled as a negative example. This smallest trigger strength is what they call the budget curve.

Critical strengths

For me, the theoretical message is clean and easy to digest. The fact that it's easier to backdoor in the feature learning regime is rather surprising to me, and it seems the emprical evidence provided in the paper also supports this.

Critical weaknesses

You wrote in the discussion, as well as some mention of a variant of this statement in the abstract: "Our results offer a mechanism for the observation that the number of poisoned samples needed to backdoor a large model does not grow with its training corpus." But your work doesn't seem to support this. For example, the number of poisoned samples for a quadratic feature model is πN/2\pi N / 2 for NN total samples in your data, which is proportional $ N / \alpha^4$ which clearly still grows in NN.

Questions for authors

See the critical weakness section.

1 comment

Thank you for correcting this discrepancy. My score remains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.