- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a malicious user can poison a small fraction of crowdsourced fine-tuning data to amplify the extraction of previously unseen instructions contributed by benign users. The authors propose a topic-based poisoning attack that uses topic anchors to facilitate training-data extraction through black-box, output-only queries to the fine-tuned model. Experiments demonstrate that the proposed method substantially increases extraction rates under an SFT objective that computes loss over both instruction and response tokens.
Critical strengths
-
Clear motivation. The paper is well-motivated and clearly identifies an important privacy risk in crowdsourced fine-tuning, where malicious users can poison training data to facilitate the extraction of benign users' data.
-
Interesting method with realistic target-model access assumptions. The proposed poisoning-based extraction method is interesting and requires only black-box, output-only access to the deployed target model, making the target-model access assumptions realistic for practical settings.
Critical weaknesses
Performance of the extraction rate depends highly on the SFT training objective. The main experiments compute loss over both instruction and response tokens, whereas many SFT pipelines apply loss only to response tokens. As shown in Appendix I.6 (Table 16), the near-verbatim extraction gains become substantially smaller under response-only training. However, the main results emphasize large extraction improvements without adequately highlighting their dependence on the training objective. This may give readers the misleading impression that the reported gains generalize across different SFT objectives, potentially overstating the attack's effectiveness.
Unclear assumption regarding the attacker's knowledge of the topic. The proposed attack relies on a pre-defined topic taxonomy and reference datasets to select poisoning anchors. In Section 3, the authors assume that the attacker knows only the general domain of the crowdsourced data, such as math or coding. It is unclear whether the experimental setup implicitly assumes that the attacker knows the topic coverage of the benign users' fine-tuning data, and how the attack's effectiveness would change without such knowledge.
Questions for authors
-
Topic knowledge: Does the attacker need to know the range of topics covered by the target fine-tuning dataset?
-
Reference dataset coverage: To what extent do the reference datasets cover the topics present in the target fine-tuning datasets? How does incomplete coverage affect extraction performance?
-
Sensitivity to topic coverage and granularity: How sensitive is the attack's effectiveness to mismatches between the reference and target topic distributions, as well as to different levels of granularity in the topic taxonomy?
6 comments
Q3. Sensitivity to topic coverage and granularity
Sensitivity to topic coverage and granularity: How sensitive is the attack's effectiveness to mismatches between the reference and target topic distributions, as well as to different levels of granularity in the topic taxonomy?
As for topic coverage, we separate coverage in the poisoning set from coverage in the extraction queries. Appendix F examines poisoning-topic coverage by varying both the number of poisoning anchors and the topics selected. The experiment in Q2 additionally tests a reference–target mismatch in which the reference and poisoning data cover only math, while the target data contains both math and coding records.
As for extraction coverage, we evaluate a condition that randomly retains half of the available extraction anchors while keeping the total query budget fixed. So each anchor will receive more queries, but we will have fewer topics to query. We evaluate recovery against the entire target training dataset, including records associated with omitted topics.
OpenMathInstruct / Qwen2.5-7B-It
| Metric | Half | Original |
|---|---|---|
| Near-verbatim ER | 6.94 | 6.48 |
| Semantic ER | 8.58 | 7.25 |
| BLEU max | 73.0 | 70.8 |
| BLEU top-10 | 52.9 | 41.1 |
| Topic coverage (%) | 41.46 | 42.44 |
AceReason / Llama-3.1-8B-It
| Metric | Half | Original |
|---|---|---|
| Near-verbatim ER | 5.22 | 6.07 |
| Semantic ER | 7.73 | 10.10 |
| BLEU max | 64.8 | 62.9 |
| BLEU top-10 | 45.6 | 42.7 |
| Topic coverage (%) | 23.08 | 25.72 |
Under matched total query budgets, halving the extraction-topic set reduces recovered-topic coverage by 0.98 percentage points on OpenMathInstruct and 2.64 points on AceReason. When we reduce extraction query topics by half, extraction increases on OpenMathInstruct but decreases on AceReason. Overall, the attack remains effective even with half the extraction topics removed.
Reference data guide poisoning-topic selection, while the extraction taxonomy is constructed independently of both datasets. The math-only-reference experiment in Q2 therefore tests the effect of missing reference topics on poisoning design; the extraction-subset experiment separately tests incomplete extraction-topic coverage.
As for topic granularity, we compare extraction using the generic domain anchor "math" with the regular topic taxonomy (domain/topic/subtopic) on OpenMathInstruct / Qwen2.5-7B-It. We report extraction rates and extracted-topic coverage below.
| Metric | Generic ("math") | Regular |
|---|---|---|
| Near-verbatim ER, SFT (%) | 1.48 | 1.84 |
| Near-verbatim ER, Poison (%) | 5.26 | 6.48 |
| Semantic ER, SFT (%) | 1.86 | 2.73 |
| Semantic ER, Poison (%) | 7.15 | 7.25 |
| Extracted-topic coverage | 35.12% | 42.44% |
The generic “math” anchor still enables effective extraction, achieving 5.26% near-verbatim ER compared with 6.48% using finer-grained anchors. Finer-grained querying increases topic coverage among near-verbatim recoveries from 35.12% to 42.44%. This shows that detailed extraction cues improve near-verbatim recovery and topic coverage, but are not necessary for the attack to remain effective. We have updated our manuscript and included the discussion of topic coverage and granularity in Appendix J.
Q4. Extraction under response-only loss
Performance of the extraction rate depends highly on the SFT training objective.
We decided to compute loss over both instruction and response tokens for our primary experiments following Zhang et al. We also evaluate response-only loss, which masks instruction tokens, in Appendix I.6. We have expanded that evaluation to include both direct-submission and hosted-interaction poisoning at P=100 and P=200, with results across three training seeds. In both evaluated configurations, mean near-verbatim extraction remains higher under poisoning than under benign SFT, even with response-only loss.
OpenMathInstruct / Qwen2.5-7B-It
| Metric | SFT* | Direct P=100 | Direct P=200 | Hosted P=100 | Hosted P=200 |
|---|---|---|---|---|---|
| Near-verbatim ER | 0.60 ± 0.51 | 1.98 ± 0.25 | 2.20 ± 0.16 | 1.64 ± 0.05 | 1.50 ± 0.08 |
| Semantic ER | 0.83 ± 0.84 | 3.01 ± 0.26 | 2.92 ± 0.09 | 2.92 ± 0.10 | 2.42 ± 0.14 |
| Max BLEU | 30.4 ± 6.2 | 46.0 ± 0.9 | 48.7 ± 1.0 | 42.8 ± 0.5 | 43.0 ± 0.3 |
| Top-10 BLEU | 22.2 ± 5.7 | 27.1 ± 0.6 | 28.2 ± 0.4 | 31.1 ± 0.1 | 32.0 ± 0.2 |
AceReason / Llama-3.1-8B-It
| Metric | SFT* | Direct P=100 | Direct P=200 | Hosted P=100 | Hosted P=200 |
|---|---|---|---|---|---|
| Near-verbatim ER | 0.24 ± 0.32 | 0.84 ± 0.11 | 2.64 ± 0.11 | 0.73 ± 0.05 | 0.69 ± 0.06 |
| Semantic ER | 0.65 ± 1.06 | 2.78 ± 0.22 | 3.51 ± 0.07 | 1.05 ± 0.04 | 1.02 ± 0.06 |
| Max BLEU | 19.0 ± 3.3 | 43.1 ± 1.2 | 50.4 ± 1.0 | 39.2 ± 1.8 | 42.2 ± 0.6 |
| Top-10 BLEU | 12.2 ± 0.3 | 28.4 ± 0.5 | 35.0 ± 1.1 | 24.3 ± 0.7 | 27.2 ± 0.3 |
All conditions use n=3 runs with training seeds 42, 43, and 44.
SFT* means the response-only loss fine-tuned model without poisoning. ± means sample standard deviation.
Both poisoning variants improve extraction performance over benign SFT across all reported metrics in these two configurations. Increasing the direct-submission budget from P=100 to P=200 raises near-verbatim extraction in both configurations, most clearly for Llama-3.1-8B on AceReason: 0.24% under SFT, 0.84% at P=100, and 2.64% at P=200. Other metrics do not always improve with a larger budget. Hosted poisoning yields lower extraction rates than direct submission, but it still exceeds benign SFT by approximately 2.5–3.0 times on near-verbatim ER. We have added this expanded experiment under Section 5.8 in our manuscript.
We hope our response and additional experimental results address your concerns. If our response addresses your questions, we kindly ask that you consider re-evaluating your score. We are more than happy to provide additional details if any further questions remain.
We thank the reviewer for the detailed feedback. We address each question below.
Summary of our response
Q1. Topic knowledge
No. The attacker knows the broad application domain, such as math or coding, but does not need to know the specific topics covered by the target fine-tuning dataset. We constructed a taxonomy of 416 math and coding topic anchors independently of both the target and reference datasets. The number of anchors is an experimental choice, not a requirement of the attack.
Q2. Reference-dataset coverage
Below, we first measure which topics in our independently constructed taxonomy occur in the evaluated samples from each dataset. We then measure how many target topics are represented in the corresponding reference sample: MathInstruct for OpenMathInstruct, and UltraInteract for AceReason.
Coverage of the 416-topic taxonomy
Target-topic coverage by the reference dataset
For reference dataset R and target dataset T, we measure coverage as 100 × |topics(R) ∩ topics(T)| / |topics(T)|. The denominator is the number of topics observed in the target sample.
MathInstruct covers 122 of the 140 topics observed in OpenMathInstruct (87.14%). UltraInteract covers 212 of the 306 topics observed in AceReason (69.28%), with greater coverage of coding topics than math topics. Coverage is therefore incomplete for both reference–target pairs.
To examine how incomplete coverage of reference data affects extraction, we replace UltraInteract with the math-labeled subset of MathInstruct to select poisoning topics for AceReason, whose target data contain both math and coding questions. We evaluate direct-submission poisoning on Llama-3.1-8B with P=100, using exclusively math poisoning topics. We compare this condition with the standard mixed-domain poisoning (currently presented in our main evaluation) while keeping the poisoning and total extraction budgets fixed.
Llama-3.1-8B-It / AceReason: mixed-domain vs. math-only poisoning
Mixed-domain poisoning extracts more, as expected when the poisons cover both domains of the target data. However, math-only poisoning still clearly exceeds benign SFT (e.g., near-verbatim ER 1.90 → 4.27, vs. 6.07 with mixed-domain poisons).