- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper motivates to select a representative poisoning set, which is often an overlooked factor in LLM backdoor evaluation. The authors propose SAILS attack that learns to select more effective poison sets, outperforming influence-based baselines and demonstrating that random selection can substantially underestimate backdoor vulnerability.
Critical strengths
- Selecting an effective poisoning set is a better attack approach.
I appreciate the idea of selecting a representative poisoning set to better uncover the LLM's vulerability against data-poisoning backdoors.
- The evaluation is comprehensive.
The experiments with three regimes and across different models comprehensively show the effectiveness of proposed SAILS attack.
Critical weaknesses
- The formulation of the loss function of scorer is unclear
In the discussion of non-additivity at lines 143–144, the meanings of , , and are not sufficiently explained. In particular, each sample is marked with the subscript . Given the formulation at lines 143-144, should have multiple items after to indicate the combinatorial influence between samples and , which are unfortunately hidden in the dots .
- Insufficient detail on candidate-pool construction and subset generation.
Section G.3 compares LM-based generation with pool-based selection, indicating a high importance of the candidate pool construction. However, the pool’s source data, preprocessing, filtering, and transformation into poison candidates are unclear. Meanwhile, Section 2 should also specify the default SAILS setting, distinguishing candidate-pool construction from the procedure used to propose subsets from that pool.
- The price of poisoning is actually larger than described.
The final poisoning ratio measures the amount of poisoned samples, but does not represent the real resources required to identify an effective poison set. Given that SAILS additionally relies on repetitively fine-tuning and evaluating the oracle calls for scorer training and candidate auditing, the success of the SAILS attack relies on the guidance knowledge obtained by accessing to the actual victim model, which, thus, requires a large preparation cost, including initial oracle labels and iterative audits. Moreover, the threat model allows white-box access to the victim model's gradients, exposing the full model knowledge to the attacker. Taken together, this attacker-favoring setting is overly strong and significantly lowers the practicality in comparison to related attacks that modify only the training data without querying the victim model.
Questions for authors
-
What is the reason for adding a constant-like value to ? Does the constant remain the same for the entire subset or should it be written as for an individual sample in ?
-
SAILS poisoning relies on access to the oracle LLM that is fixed during the generation of poisoning data. In the case that the LLM is kept updated via online continuous learning, does the poisoning remain effective?
Summary of revisions and reviewer discussion
We thank the reviewers and Area Chair for their time and consideration. As the discussion period concludes, we wanted to briefly summarize the revisions and subsequent reviewer feedback.
In response to the reviews, we expanded the evaluation with new experiments comparing uncertainty-aware and diversity-based acquisition strategies, analyses of the relationship between triggered loss and attack success, and transfer experiments using a single source-selected poison set across model families. We also clarified the WebShop evaluation by confirming completed target-item purchases, provided detailed accounting of the API-only attack's oracle budget and preparation costs, and expanded our analysis of set interactions and backdoor persistence under continued finetuning. Additionally, we strengthened reproducibility by updating the code release, providing the TRAK + representer baseline pipeline, and clarifying the scorer formulation, candidate-pool construction, and evaluation protocols.
We appreciate the positive feedback following these additions. Reviewer 581E increased their rating from 2 to 3, confirming that our responses addressed their concerns. Reviewer RTGw also confirmed that our revisions and clarifications addressed their main concerns, while maintaining their rating of 3. Reviewer sdhu also rated the paper 3.
We hope these revisions and the discussion help inform the Area Chair's final assessment, and we thank everyone again for their constructive feedback.