- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the detectability and robustness of encrypted neural backdoors. The authors propose DIABLO, a statistical detector based on parameter distributions, and PAWS, a randomized parameter assignment scheme to evade detection while preserving Boolean functionality. They further investigate tree-based XOR implementations to improve robustness against parameter perturbations.
Critical strengths
1.The paper addresses an interesting security problem. Studying both encrypted backdoor detection and adaptive evasion provides useful insights into the limitations of parameter-based detection. 2.PAWS provides an interesting approach to preserving Boolean functionality while randomizing parameter values. The paper is generally well organized and easy to follow.
Critical weaknesses
Limited practical relevance and insufficient experimental validation. The experiments compare standalone compiled backdoor MLPs with benign Transformer models. No backdoor is actually integrated into a pretrained LLM. Consequently, the reported detection and evasion results do not establish their effectiveness on real backdoored LLMs. The robustness evaluation is also limited to reduced Keccak configurations, a single trigger/payload pair, and Gaussian noise, without evaluating the impact on normal model performance.
Weak novelty in detection and insufficient support for key claims. DIABLO relies on simple parameter uniqueness and KS-statistic thresholds. PAWS largely evades this detector by increasing parameter diversity, which directly defeats the detector's predefined uniqueness criterion. This result is somewhat expected, and no stronger detection baselines are evaluated. Moreover, Theorem 3.5 claims distribution alignment without sufficient theoretical justification. PAWS uses sign-constrained sampling and gate-dependent biases, which do not generally guarantee that the resulting parameter mean and standard deviation approximate those of the target model.
Questions for authors
1.Can the authors demonstrate DIABLO and PAWS on an actual backdoored LLM rather than isolated MLP modules? 2.How does PAWS perform against alternative statistical or learning-based anomaly detectors? How does DIABLO handle quantized or sparse benign models? 3.Can the authors provide a rigorous justification for the distribution alignment claimed in Theorem 3.5?