acceptodds
Under review as a conference paper at ICLR 2027

The Privacy Fallacy of Crowdsourced Fine-tuning: Extracting Proprietary Data via Topic-Based Poisoning

Abstract

Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks. Crowdsourcing user conversations is an established approach to collecting SFT data at scale while reducing the need for costly manual annotation. However, it also allows untrusted users to contribute data to the fine-tuning pipeline. We investigate an underexplored privacy risk arising from this setting: can a malicious user poison a small fraction of the crowdsourced data to amplify extraction of previously unseen instructions contributed by other users? We show that this is possible using only black-box, output-only access to the deployed model. Experiments across four models and two datasets demonstrate substantial increases in training-data extraction: with only 50 poisoned examples, near-verbatim extraction reaches the rate without poisoning for Qwen2.5-14B on OpenMathInstruct and for Llama-3.1-8B on AceReason. Data filtering also proves largely ineffective in detecting poisoned samples: even the best-performing method achieves an F1 score of only 0.378 and leaves the majority of poisoned samples undetected. These findings demonstrate that seemingly benign crowdsourced contributions can amplify leakage of other records while remaining difficult to identify through data filtering.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
training data extraction, poisoning, supervised fine-tuning, memorization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SYqsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether a malicious user can poison a small fraction of crowdsourced fine-tuning data to amplify the extraction of previously unseen instructions contributed by benign users. The authors propose a topic-based poisoning attack that uses topic anchors to facilitate training-data extraction through black-box, output-only queries to the fine-tuned model. Experiments demonstrate that the proposed method substantially increases extraction rates under an SFT objective that computes loss over both instruction and response tokens.

Critical strengths
  1. Clear motivation. The paper is well-motivated and clearly identifies an important privacy risk in crowdsourced fine-tuning, where malicious users can poison training data to facilitate the extraction of benign users' data.

  2. Interesting method with realistic target-model access assumptions. The proposed poisoning-based extraction method is interesting and requires only black-box, output-only access to the deployed target model, making the target-model access assumptions realistic for practical settings.

Critical weaknesses

Performance of the extraction rate depends highly on the SFT training objective. The main experiments compute loss over both instruction and response tokens, whereas many SFT pipelines apply loss only to response tokens. As shown in Appendix I.6 (Table 16), the near-verbatim extraction gains become substantially smaller under response-only training. However, the main results emphasize large extraction improvements without adequately highlighting their dependence on the training objective. This may give readers the misleading impression that the reported gains generalize across different SFT objectives, potentially overstating the attack's effectiveness.

Unclear assumption regarding the attacker's knowledge of the topic. The proposed attack relies on a pre-defined topic taxonomy and reference datasets to select poisoning anchors. In Section 3, the authors assume that the attacker knows only the general domain of the crowdsourced data, such as math or coding. It is unclear whether the experimental setup implicitly assumes that the attacker knows the topic coverage of the benign users' fine-tuning data, and how the attack's effectiveness would change without such knowledge.

Questions for authors
  1. Topic knowledge: Does the attacker need to know the range of topics covered by the target fine-tuning dataset?

  2. Reference dataset coverage: To what extent do the reference datasets cover the topics present in the target fine-tuning datasets? How does incomplete coverage affect extraction performance?

  3. Sensitivity to topic coverage and granularity: How sensitive is the attack's effectiveness to mismatches between the reference and target topic distributions, as well as to different levels of granularity in the topic taxonomy?

6 comments

We thank the reviewer for the detailed feedback. We address each question below.

Summary of our response

  • Attacker knowledge: The attacker only needs the domain to construct extraction queries (topics) without access to target or reference data.
  • Topic coverage/Granularity: Fine-grained extraction anchors increase recovered-topic coverage from 35.12% to 42.44% compared with the generic “math” cue. However, even without fine-grained topic taxonomy, poisoning remains effective. (Appendix J)
  • Reference-data mismatch: When the attacker’s reference data cover only math but the target data (AceReason) contain both math and coding, poisoning still exceeds benign SFT (near-verbatim ER 4.27 vs. 1.90), though below mixed-domain poisoning (6.07). (Appendix J)
  • Response-only loss: Both direct and hosted poisoning still increase extraction. (Section 5.8, Appendix L)

Q1. Topic knowledge

Topic knowledge: Does the attacker need to know the range of topics covered by the target fine-tuning dataset?

No. The attacker knows the broad application domain, such as math or coding, but does not need to know the specific topics covered by the target fine-tuning dataset. We constructed a taxonomy of 416 math and coding topic anchors independently of both the target and reference datasets. The number of anchors is an experimental choice, not a requirement of the attack.

Q2. Reference-dataset coverage

Reference dataset coverage: To what extent do the reference datasets cover the topics present in the target fine-tuning datasets? How does incomplete coverage affect extraction performance?

Below, we first measure which topics in our independently constructed taxonomy occur in the evaluated samples from each dataset. We then measure how many target topics are represented in the corresponding reference sample: MathInstruct for OpenMathInstruct, and UltraInteract for AceReason.

Coverage of the 416-topic taxonomy

DatasetRowsTopics covered / 416CoverageMath / 205Code / 211
OpenMathInstruct5,040140/41633.65%140/205 (68.29%)0/211 (0.00%)
AceReason5,040306/41673.56%158/205 (77.07%)148/211 (70.14%)
MathInstruct5,042284/41668.27%169/205 (82.44%)115/211 (54.50%)
UltraInteract5,040251/41660.34%101/205 (49.27%)150/211 (71.09%)

Target-topic coverage by the reference dataset

For reference dataset R and target dataset T, we measure coverage as 100 × |topics(R) ∩ topics(T)| / |topics(T)|. The denominator is the number of topics observed in the target sample.

Reference → targetDomainShared / target topicsTarget-topic coverageTarget topics absent from reference
MathInstruct → OpenMathInstructAll122/14087.14%18
MathInstruct → OpenMathInstructMath122/14087.14%18
MathInstruct → OpenMathInstructCode0/0N/A0
UltraInteract → AceReasonAll212/30669.28%94
UltraInteract → AceReasonMath90/15856.96%68
UltraInteract → AceReasonCode122/14882.43%26

MathInstruct covers 122 of the 140 topics observed in OpenMathInstruct (87.14%). UltraInteract covers 212 of the 306 topics observed in AceReason (69.28%), with greater coverage of coding topics than math topics. Coverage is therefore incomplete for both reference–target pairs.

To examine how incomplete coverage of reference data affects extraction, we replace UltraInteract with the math-labeled subset of MathInstruct to select poisoning topics for AceReason, whose target data contain both math and coding questions. We evaluate direct-submission poisoning on Llama-3.1-8B with P=100, using exclusively math poisoning topics. We compare this condition with the standard mixed-domain poisoning (currently presented in our main evaluation) while keeping the poisoning and total extraction budgets fixed.

Llama-3.1-8B-It / AceReason: mixed-domain vs. math-only poisoning

MetricSFTMixed-domain (main)Math-only
Near-verbatim ER1.906.074.27
Semantic ER3.7910.107.09
BLEU max52.062.958.4
BLEU top-1034.642.739.6

Mixed-domain poisoning extracts more, as expected when the poisons cover both domains of the target data. However, math-only poisoning still clearly exceeds benign SFT (e.g., near-verbatim ER 1.90 → 4.27, vs. 6.07 with mixed-domain poisons).

Q3. Sensitivity to topic coverage and granularity

Sensitivity to topic coverage and granularity: How sensitive is the attack's effectiveness to mismatches between the reference and target topic distributions, as well as to different levels of granularity in the topic taxonomy?

As for topic coverage, we separate coverage in the poisoning set from coverage in the extraction queries. Appendix F examines poisoning-topic coverage by varying both the number of poisoning anchors and the topics selected. The experiment in Q2 additionally tests a reference–target mismatch in which the reference and poisoning data cover only math, while the target data contains both math and coding records.

As for extraction coverage, we evaluate a condition that randomly retains half of the available extraction anchors while keeping the total query budget fixed. So each anchor will receive more queries, but we will have fewer topics to query. We evaluate recovery against the entire target training dataset, including records associated with omitted topics.

OpenMathInstruct / Qwen2.5-7B-It

MetricHalfOriginal
Near-verbatim ER6.946.48
Semantic ER8.587.25
BLEU max73.070.8
BLEU top-1052.941.1
Topic coverage (%)41.4642.44

AceReason / Llama-3.1-8B-It

MetricHalfOriginal
Near-verbatim ER5.226.07
Semantic ER7.7310.10
BLEU max64.862.9
BLEU top-1045.642.7
Topic coverage (%)23.0825.72

Under matched total query budgets, halving the extraction-topic set reduces recovered-topic coverage by 0.98 percentage points on OpenMathInstruct and 2.64 points on AceReason. When we reduce extraction query topics by half, extraction increases on OpenMathInstruct but decreases on AceReason. Overall, the attack remains effective even with half the extraction topics removed.

Reference data guide poisoning-topic selection, while the extraction taxonomy is constructed independently of both datasets. The math-only-reference experiment in Q2 therefore tests the effect of missing reference topics on poisoning design; the extraction-subset experiment separately tests incomplete extraction-topic coverage.

As for topic granularity, we compare extraction using the generic domain anchor "math" with the regular topic taxonomy (domain/topic/subtopic) on OpenMathInstruct / Qwen2.5-7B-It. We report extraction rates and extracted-topic coverage below.

MetricGeneric ("math")Regular
Near-verbatim ER, SFT (%)1.481.84
Near-verbatim ER, Poison (%)5.266.48
Semantic ER, SFT (%)1.862.73
Semantic ER, Poison (%)7.157.25
Extracted-topic coverage35.12%42.44%

The generic “math” anchor still enables effective extraction, achieving 5.26% near-verbatim ER compared with 6.48% using finer-grained anchors. Finer-grained querying increases topic coverage among near-verbatim recoveries from 35.12% to 42.44%. This shows that detailed extraction cues improve near-verbatim recovery and topic coverage, but are not necessary for the attack to remain effective. We have updated our manuscript and included the discussion of topic coverage and granularity in Appendix J.

Q4. Extraction under response-only loss

Performance of the extraction rate depends highly on the SFT training objective.

We decided to compute loss over both instruction and response tokens for our primary experiments following Zhang et al. We also evaluate response-only loss, which masks instruction tokens, in Appendix I.6. We have expanded that evaluation to include both direct-submission and hosted-interaction poisoning at P=100 and P=200, with results across three training seeds. In both evaluated configurations, mean near-verbatim extraction remains higher under poisoning than under benign SFT, even with response-only loss.

OpenMathInstruct / Qwen2.5-7B-It

MetricSFT*Direct P=100Direct P=200Hosted P=100Hosted P=200
Near-verbatim ER0.60 ± 0.511.98 ± 0.252.20 ± 0.161.64 ± 0.051.50 ± 0.08
Semantic ER0.83 ± 0.843.01 ± 0.262.92 ± 0.092.92 ± 0.102.42 ± 0.14
Max BLEU30.4 ± 6.246.0 ± 0.948.7 ± 1.042.8 ± 0.543.0 ± 0.3
Top-10 BLEU22.2 ± 5.727.1 ± 0.628.2 ± 0.431.1 ± 0.132.0 ± 0.2

AceReason / Llama-3.1-8B-It

MetricSFT*Direct P=100Direct P=200Hosted P=100Hosted P=200
Near-verbatim ER0.24 ± 0.320.84 ± 0.112.64 ± 0.110.73 ± 0.050.69 ± 0.06
Semantic ER0.65 ± 1.062.78 ± 0.223.51 ± 0.071.05 ± 0.041.02 ± 0.06
Max BLEU19.0 ± 3.343.1 ± 1.250.4 ± 1.039.2 ± 1.842.2 ± 0.6
Top-10 BLEU12.2 ± 0.328.4 ± 0.535.0 ± 1.124.3 ± 0.727.2 ± 0.3

All conditions use n=3 runs with training seeds 42, 43, and 44.

SFT* means the response-only loss fine-tuned model without poisoning. ± means sample standard deviation.

Both poisoning variants improve extraction performance over benign SFT across all reported metrics in these two configurations. Increasing the direct-submission budget from P=100 to P=200 raises near-verbatim extraction in both configurations, most clearly for Llama-3.1-8B on AceReason: 0.24% under SFT, 0.84% at P=100, and 2.64% at P=200. Other metrics do not always improve with a larger budget. Hosted poisoning yields lower extraction rates than direct submission, but it still exceeds benign SFT by approximately 2.5–3.0 times on near-verbatim ER. We have added this expanded experiment under Section 5.8 in our manuscript.

We hope our response and additional experimental results address your concerns. If our response addresses your questions, we kindly ask that you consider re-evaluating your score. We are more than happy to provide additional details if any further questions remain.

Reviewer_xfvmOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Many model providers use user-contributed conversations or Q&A pairs for SFT. This paper demonstrates that a malicious user can cause a deployed model to more easily "spit out" training data contributed by other users by injecting just a tiny amount of seemingly normal-looking data (approximately 50–100 examples, or about 1–2% of the dataset). The attacker only requires black-box access, meaning they can only observe the text output by the model.

Critical strengths
  1. It's a realistic, weaker threat model.
  2. The experimental coverage is broad.
Critical weaknesses
  1. There is no non-member control. The paper never measures how often the poisoned model "matches" held-out questions from the same dataset that were not in the training split. You may insert unique canaries and threshold sweeps.
  2. Some obvious targeted defenses, prompt masking, DP fine-tuning, deployment-time monitoring are omited and never evaluated. The title and abstract emphasize user privacy and proprietary data, but all experiments use public, largely synthetic math and code datasets. There is no PII, no user-specific conversational data, and no canaries.
Questions for authors
  1. What is the matching rate against held-out non-member questions from the same datasets, for both benign SFT and poisoned models? How do the results change under threshold sweeps?
  2. How was the MP baseline reproduced? Did it use the same query budget and adaptive querying? Why is it sometimes below the base model?
  3. How does a cross-field duplicate filter perform (response ≈ another record's instruction)? How does a template or intent filter perform in the hosted setting?
  4. Figure 5b reports 129/4,940 ≈ 2.6% reconstructed for Qwen2.5-7B/OMI. This does not match Table 1 (6.48%), Table 10 (3.62–6.48%), or Figure 5c (5.04%). What setting was used?
  5. Can you report variance across fine-tuning seeds and sampling runs for Tables 1, 10, and 13?
7 comments

We thank the reviewer for the detailed feedback. We address each question below.

Summary of our response

  • Member versus non-member recovery: Extraction rates and other metrics are higher under members than under held-out non-members in nearly all comparisons, and the member–non-member gap widens after poisoning in 26 of 32 comparisons. We also conduct threshold sweeps. (Section 5.7, Appendix K)
  • MP baseline: We use the same total query budget and MP's original querying procedure, without our adaptive querying.
  • Filtering defenses: We clarify the attack knowledge assumed by targeted filters and compare template/wording-based filtering vs intent-based filtering (such as LLM-judge).
  • Figure consistency: We clarify the older experimental setting, and Figure 5b has been regenerated using the Table 1 results.
  • Training/generation variability: We report three-seed results for two model–dataset combinations, with further evaluations in progress. (Appendix L)
  • Confidentiality framing: We clarify that public datasets serve as reproducible proxies for confidential fine-tuning instructions.
  • Additional defenses: We point to prompt-masking results (Appendix I.6, Section 5.8) and will follow up with DP-tuning experiments.

Q1. Member recovery and threshold sweeps

What is the matching rate against held-out non-member questions from the same datasets, for both benign SFT and poisoned models? How do the results change under threshold sweeps?

We compare extraction performance against member (Mem) and held-out non-member (Non-mem) instructions for both benign baseline SFT and poisoned models with P=100. Bold indicates the higher value within each Mem/Non-mem pair.

Across both datasets and models, members remain more extractable than non-members in every metric-model pair under poisoning. While poisoning increases extraction for both member and non-member instructions, the member-non-member gap also widens under poisoning in most settings (26 out of 32). For instance, the near-verbatim ER gap grows from 1.19 to 3.67 for Qwen2.5-14B on OpenMathInstruct and 0.96 to 4.19 for Llama-3.1-8B on AceReason. These results show that poisoning also amplifies leakage of specific training records beyond population-level learning.

OpenMathInstruct

MetricModelSFT MemSFT Non-memPoison MemPoison Non-mem
Near-verbatim ERQwen2.5-7B1.840.996.483.18
Llama-3.1-8B2.310.992.771.19
Qwen2.5-14B2.381.198.444.77
Mistral-NeMo-12B1.850.804.471.99
Semantic ERQwen2.5-7B2.733.987.254.17
Llama-3.1-8B4.153.385.263.78
Qwen2.5-14B3.682.7810.085.57
Mistral-NeMo-12B2.682.586.583.18
BLEU maxQwen2.5-7B43.730.370.837.0
Llama-3.1-8B56.136.959.039.9
Qwen2.5-14B44.229.677.839.0
Mistral-NeMo-12B46.630.964.040.3
BLEU top-10Qwen2.5-7B32.822.841.122.9
Llama-3.1-8B39.126.341.027.4
Qwen2.5-14B31.421.649.926.2
Mistral-NeMo-12B33.023.044.128.0

AceReason

MetricModelSFT MemSFT Non-memPoison MemPoison Non-mem
Near-verbatim ERQwen2.5-7B1.191.511.281.13
Llama-3.1-8B1.900.946.071.88
Qwen2.5-14B1.841.512.592.26
Mistral-NeMo-12B1.210.565.531.69
Semantic ERQwen2.5-7B2.331.514.333.58
Llama-3.1-8B3.791.5110.107.34
Qwen2.5-14B3.121.698.086.78
Mistral-NeMo-12B2.741.329.626.59
BLEU maxQwen2.5-7B30.518.841.731.5
Llama-3.1-8B52.035.162.941.5
Qwen2.5-14B38.525.360.038.0
Mistral-NeMo-12B38.026.365.240.7
BLEU top-10Qwen2.5-7B21.013.426.320.9
Llama-3.1-8B34.626.442.731.0
Qwen2.5-14B25.917.634.825.1
Mistral-NeMo-12B26.718.743.429.8

We also add the results for threshold sweeps below.

Near-verbatim ER under threshold sweep (Mem / Non-mem). Default threshold = 0.80 for OpenMathInstruct and 0.75 for AceReason.

DatasetModelSettingτ=0.95τ=0.85τ=0.75
OpenMathInstructQwen2.5-7BSFT0.28 / 0.401.19 / 0.992.63 / 1.99
Poison3.08 / 0.805.49 / 2.587.53 / 4.77
Llama-3.1-8BSFT0.45 / 0.001.56 / 0.603.14 / 2.19
Poison0.43 / 0.001.76 / 0.804.27 / 1.59
Qwen2.5-14BSFT0.81 / 0.401.78 / 0.993.04 / 1.79
Poison4.98 / 0.997.47 / 3.7810.02 / 7.75
Mistral-NeMo-12BSFT0.18 / 0.001.19 / 0.602.56 / 1.59
Poison0.97 / 0.002.77 / 1.196.17 / 2.98
AceReasonQwen2.5-7BSFT0.24 / 0.000.49 / 0.381.19 / 1.51
Poison0.26 / 0.190.45 / 0.191.28 / 1.13
Llama-3.1-8BSFT0.10 / 0.000.55 / 0.191.90 / 0.94
Poison0.99 / 0.002.69 / 0.386.07 / 1.88
Qwen2.5-14BSFT0.40 / 0.000.95 / 0.381.84 / 1.51
Poison0.61 / 0.561.23 / 0.562.59 / 2.26
Mistral-NeMo-12BSFT0.06 / 0.000.30 / 0.001.21 / 0.56
Poison1.07 / 0.002.45 / 0.385.53 / 1.69

Under poisoning, members exceed non-members at every threshold (24/24 cells), and poisoning increases member extraction over SFT in 22 of 24 cells. We have updated our manuscript and included a member vs. non-member comparison and threshold sweep in Section 5.7 and Appendix K, respectively.

Q2. MP baseline reproduction

How was the MP baseline reproduced? Did it use the same query budget and adaptive querying? Why is it sometimes below the base model?

We evaluated MP with the same total query budget: a total of 20,500 completions for OpenMathInstruct and 41,600 for AceReason. We reused the opening words released in Zhang et al.’s GitHub repository, and we followed their original querying procedure without adding our adaptive querying.

Two factors may explain MP’s lower extraction performance. Table 9 shows that topic-based queries generally outperform opening-word queries, including without poisoning. MP implants its backdoor in its instruction-following data before downstream fine-tuning, so the later SFT data are unavailable for constructing topic-to-question training pairs. Thus, MP uses opening words instead, for both poisoning and extraction. This difference in extraction prompts could explain why MP sometimes falls below the Base and SFT models, which we query with topic prompts.

A second plausible factor is that MP relies on extraction behavior learned during the earlier instruction-following stage to transfer to subsequent downstream fine-tuning. This transfer may be generally less effective than introducing poisoning during downstream fine-tuning (our attack).

Q3. Filtering defenses

How does a cross-field duplicate filter perform (response ≈ another record's instruction)? How does a template or intent filter perform in the hosted setting?

The effectiveness of these targeted defenses depends on our assumption about which features of the attack design are known to the defender.

For example, template filtering is highly effective if the defender knows our exact question-generation wording. It would achieve 100% precision and recall on our current dataset since every poisoning request contains that template/wording, and no benign instruction does. Cross-field filtering similarly would achieve 100% recall for the current poisoning construction because each generated question is reused exactly as another instruction. If an attacker is aware of such a defense, then they could avoid these exact matches by rephrasing the reused question. In that case, detecting those variations would require broader matching, for example through an LLM intent judge or embedding-based cross-field similarity. We are evaluating a variant that paraphrases the questions in the question-generation responses while keeping the requests unchanged. We will follow up with experimental results on its poisoning effectiveness and cross-field detection.

As for intent-based filtering, our earlier experiments also found effective poisoning with the static trigger “Q” rather than the interpretable request “Please provide a question about…”, showing that poisoning does not require an explicit question-generation phrase. Based on this, we also believe that intent-based defenses need careful evaluation beyond the particular wording used in our main experiments.

Our LLM judge used in Section 6 already includes task alignment as one of its filtering criteria. To further evaluate intent-based filtering under the current wording, we modified its prompt to focus exclusively on identifying question-generation requests. The modified prompt states: “Return NO if the input primarily asks the assistant to create, retrieve, or reproduce standalone mathematical or coding questions or problems. Return YES otherwise…[abbreviated]” However, our experiment using Qwen2.5-72B-Instruct and direct-submission poisoning with this modified prompt filtered out 5129 out of 5138 samples. It catches 94 of 98 poisoned examples (96 % recall; we deduplicated poisoned samples before filtering), but its precision (1.8%) is essentially equal to the poison base rate (1.9%), so this is no better than random selection.

We also repeated the same filtering in the hosted-interaction setting, and the result remains similar: it achieves a precision of 1.84% and recall of 100%, rejecting 99.9% of all examples. While a stronger model or refined prompt may improve filtering, this high false-positive rate highlights the challenge of calibrating intent-based filtering to identify the poisoned exchanges without discarding valid samples.

Reviewer_MGx9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether malicious contributors can poison a small fraction of crowdsourced supervised fine-tuning data to amplify the extraction of instructions contributed by other users. The attacker cannot modify the base model, does not know other users’ training records, and receives only generated text from the deployed model.

The proposed method adds topic-conditioned question-generation exchanges alongside ordinary question-answer contributions. The attacker selects poisoning topics using reference data and employs surrogate-model likelihood scores to allocate extraction queries adaptively. The paper considers both direct submission, where attackers control questions and responses, and hosted interaction, where a chatbot generates the responses.

Experiments across four target models and two main datasets demonstrate increased extraction under small poisoning budgets. Additional analyses examine alternative attacks, poisoning budgets, data filtering, dataset sizes, query allocation, downstream utility, and response-only training loss.

Critical strengths
  1. The paper demonstrates a meaningful privacy vulnerability under restricted attacker capabilities. The attacker influences only a small fraction of downstream contributions, starts from a benign base model, and extracts through an output-only interface. This makes the result consequential beyond settings that require a malicious model provider or access to target-model logits. The hosted-interaction variant further strengthens the contribution by showing that direct control over contributed responses is not necessary for increased extraction.

  2. The attack is simple and supported by a substantial empirical investigation. Topic-conditioned question-generation examples provide an intuitive mechanism for making training instructions more accessible through ordinary queries. Experiments across four models and two main datasets, together with analyses of poisoning budgets, topic selection, adaptive querying, filtering, and prompt masking, provide a reasonably broad characterization of the attack. Excluding attacker-contributed questions from evaluation is an important methodological strength.

Critical weaknesses
  1. The extraction metrics need stronger validation of record-specific recovery. Near-verbatim extraction uses manually selected normalized Levenshtein thresholds, while semantic extraction permits changes in numerical values. In mathematical and coding domains, such matches can sometimes reflect common problem templates rather than recovery of a particular training record. The base-model and benign-SFT comparisons support the reported amplification, but distribution-matched nonmember controls, exact-match results, and a systematic human audit would strengthen the privacy interpretation. I view this as a limitation of the estimated leakage magnitude rather than evidence that the poisoning effect is absent.

  2. The strongest results depend on the fine-tuning loss setting. Under response-only loss, Table 16 shows considerably smaller and less consistent near-verbatim extraction gains than the primary all-token-loss experiments. The paper appropriately includes this analysis, but the main conclusions should more prominently distinguish these settings. Repeated runs would also help establish whether the remaining small improvements are robust. This limits the generality of the headline amplification results, although the primary setting still demonstrates a concrete vulnerability.

Questions for authors

The following clarification requests could materially affect my assessment:

  1. Do the poisoning gains persist under exact matching and stricter near-verbatim thresholds? Can you provide a human-audited estimate of record-specific recovery or a comparison against distribution-matched questions excluded from fine-tuning?

  2. How stable are the response-only-loss results across training seeds and independently sampled poisoning sets? Does the hosted-interaction variant also increase extraction under response-only loss?

  3. What exactly does the poisoning budget P count? Please clarify whether it includes both ordinary Q&A contributions and associated question-generation exchanges, and whether attacker-owned ordinary Q&A records are included in the matched benign-SFT baseline.

  4. Section 5.6 explicitly evaluates favorable conditions using hidden-training-data similarity for instruction selection and the same base model for distillation. Can you clarify how this result should be interpreted relative to the stated attacker capabilities, and whether any performance recovery has been measured without oracle selection?

6 comments

We thank the reviewer for the detailed feedback. We address each question below.

Summary of our response

  • Stricter matching: Poisoning gains generally persist under stricter near-verbatim thresholds and exact matching after capitalization and whitespace normalization. (Appendix K)
  • Member versus non-member recovery: Extraction rates and other metrics are higher under members than held-out non-members in nearly all comparisons, and the member–non-member gap widens after poisoning in 26 of 32 comparisons. (Section 5.7)
  • Response-only loss: Both direct and hosted poisoning still increase extraction. (Section 5.8, Appendix L)
  • Distillation without oracle selection: Random and oracle instruction selection yield similar utility when instructions are distilled from the response-only loss model.

Q1. Exact and near-verbatim recovery

Do the poisoning gains persist under exact matching and stricter near-verbatim thresholds?

Below, we present the threshold sweep for near-verbatim extraction. The following result uses the same configuration as in Table 2 (P=100, all loss). The default setting is 0.75 for AceReason and 0.80 for OpenMathInstruct.

Threshold τ = 0.85 (both datasets)

DatasetModelSFTDirectHosted
OpenMathInstructQwen2.5-7B-It1.195.491.90
OpenMathInstructLlama-3.1-8B-It1.561.761.43
OpenMathInstructQwen2.5-14B-It1.787.472.64
OpenMathInstructMistral-Nemo-12B-It1.192.771.69
AceReasonQwen2.5-7B-It0.490.450.83
AceReasonLlama-3.1-8B-It0.552.690.99
AceReasonQwen2.5-14B-It0.951.231.05
AceReasonMistral-Nemo-12B-It0.302.450.54

Threshold τ = 0.95 (both datasets)

DatasetModelSFTDirectHosted
OpenMathInstructQwen2.5-7B-It0.283.080.99
OpenMathInstructLlama-3.1-8B-It0.450.430.40
OpenMathInstructQwen2.5-14B-It0.814.981.13
OpenMathInstructMistral-Nemo-12B-It0.180.970.52
AceReasonQwen2.5-7B-It0.240.260.36
AceReasonLlama-3.1-8B-It0.100.990.69
AceReasonQwen2.5-14B-It0.400.610.44
AceReasonMistral-Nemo-12B-It0.061.070.08

We also add the results under exact matching. We normalize capitalization and whitespace.

DatasetModelSFTDirectHosted
OpenMathInstructQwen2.5-7B-It0.142.230.63
OpenMathInstructLlama-3.1-8B-It0.080.140.16
OpenMathInstructQwen2.5-14B-It0.553.680.77
OpenMathInstructMistral-Nemo-12B-It0.120.470.22
AceReasonQwen2.5-7B-It0.000.040.02
AceReasonLlama-3.1-8B-It0.000.550.12
AceReasonQwen2.5-14B-It0.000.220.12
AceReasonMistral-Nemo-12B-It0.000.530.00

Poisoning gains generally persist under these stricter criteria. Although stricter matching reduces absolute extraction rates, the poisoning advantage remains in most configurations.

We have updated our manuscript and included a threshold sweep in Appendix K.

Q2. Member versus non-member recovery

Can you provide a human-audited estimate of record-specific recovery or a comparison against distribution-matched questions excluded from fine-tuning?

We compare extraction performance against member (Mem) and held-out non-member (Non-mem) instructions for both benign baseline SFT and poisoned model with P=100. Bold indicates the higher value within each Mem/Non-mem pair.

Across both datasets and models, members remain more extractable than non-members in every metric-model pair under poisoning. While poisoning increases extraction for both member and non-member instructions, the member-non-member gap also widens under poisoning in most settings (26 out of 32 comparisons). These results show that poisoning also amplifies leakage of specific training records beyond population-level learning. We have updated our manuscript and included a member vs. non-member comparison in Section 5.7.

OpenMathInstruct

MetricModelSFT MemSFT Non-memPoison MemPoison Non-mem
Near-verbatim ERQwen2.5-7B1.840.996.483.18
Llama-3.1-8B2.310.992.771.19
Qwen2.5-14B2.381.198.444.77
Mistral-NeMo-12B1.850.804.471.99
Semantic ERQwen2.5-7B2.733.987.254.17
Llama-3.1-8B4.153.385.263.78
Qwen2.5-14B3.682.7810.085.57
Mistral-NeMo-12B2.682.586.583.18
BLEU maxQwen2.5-7B43.730.370.837.0
Llama-3.1-8B56.136.959.039.9
Qwen2.5-14B44.229.677.839.0
Mistral-NeMo-12B46.630.964.040.3
BLEU top-10Qwen2.5-7B32.822.841.122.9
Llama-3.1-8B39.126.341.027.4
Qwen2.5-14B31.421.649.926.2
Mistral-NeMo-12B33.023.044.128.0

AceReason

MetricModelSFT MemSFT Non-memPoison MemPoison Non-mem
Near-verbatim ERQwen2.5-7B1.191.511.281.13
Llama-3.1-8B1.900.946.071.88
Qwen2.5-14B1.841.512.592.26
Mistral-NeMo-12B1.210.565.531.69
Semantic ERQwen2.5-7B2.331.514.333.58
Llama-3.1-8B3.791.5110.107.34
Qwen2.5-14B3.121.698.086.78
Mistral-NeMo-12B2.741.329.626.59
BLEU maxQwen2.5-7B30.518.841.731.5
Llama-3.1-8B52.035.162.941.5
Qwen2.5-14B38.525.360.038.0
Mistral-NeMo-12B38.026.365.240.7
BLEU top-10Qwen2.5-7B21.013.426.320.9
Llama-3.1-8B34.626.442.731.0
Qwen2.5-14B25.917.634.825.1
Mistral-NeMo-12B26.718.743.429.8

Q3. Response-only loss and poisoning-set variability

How stable are the response-only-loss results across training seeds and independently sampled poisoning sets? Does the hosted-interaction variant also increase extraction under response-only loss?

Below, we present the response-only results with P=100 and P=200 under both direct and hosted poisoning settings. All entries use n=3 (training seeds 42, 43, and 44). SFT* means the response-only loss fine-tuned model without poisoning. ± means sample standard deviation.

OpenMathInstruct / Qwen2.5-7B-It

MetricSFT*Direct P=100Direct P=200Hosted P=100Hosted P=200
Near-verbatim ER0.60 ± 0.511.98 ± 0.252.20 ± 0.161.64 ± 0.051.50 ± 0.08
Semantic ER0.83 ± 0.843.01 ± 0.262.92 ± 0.092.92 ± 0.102.42 ± 0.14
Max BLEU30.4 ± 6.246.0 ± 0.948.7 ± 1.042.8 ± 0.543.0 ± 0.3
Top-10 BLEU22.2 ± 5.727.1 ± 0.628.2 ± 0.431.1 ± 0.132.0 ± 0.2

AceReason / Llama-3.1-8B-It

MetricSFT*Direct P=100Direct P=200Hosted P=100Hosted P=200
Near-verbatim ER0.24 ± 0.320.84 ± 0.112.64 ± 0.110.73 ± 0.050.69 ± 0.06
Semantic ER0.65 ± 1.062.78 ± 0.223.51 ± 0.071.05 ± 0.041.02 ± 0.06
Max BLEU19.0 ± 3.343.1 ± 1.250.4 ± 1.039.2 ± 1.842.2 ± 0.6
Top-10 BLEU12.2 ± 0.328.4 ± 0.535.0 ± 1.124.3 ± 0.727.2 ± 0.3

Under response-only loss, poisoning still increases extraction over SFT across all metrics and models. Under direct poisoning, extraction grows with the poisoning budget, most clearly for Llama-3.1-8B on AceReason (near-verbatim ER 0.24 → 0.84 → 2.64). Compared to direct poisoning, hosted one is weaker, and its extraction rates do not necessarily increase from P=100 to P=200, but it still exceeds SFT by about 2.5–3× on near-verbatim ER. In general, poisoned results show low variance across three seeds. We have updated our manuscript and included response-only results in Section 5.8 and Appendix L.

As for independently sampled poisoned runs, Appendix F examines sensitivity to poisoning-topic coverage by varying both the number of poisoning topics and the topics selected. We will run a follow-up experiment under response-only loss and assess sensitivity to the particular poisoning examples using independently sampled poisoning sets while keeping the topics, poisoning budget, and training seed fixed. We will report these results in a follow-up response.

Q4. Poisoning-budget definition

What exactly does the poisoning budget P count? Please clarify whether it includes both ordinary Q&A contributions and associated question-generation exchanges, and whether attacker-owned ordinary Q&A records are included in the matched benign-SFT baseline.

The poisoning budget P counts pairs, each consisting of one ordinary Q&A contribution and its associated question-generation exchange. In direct submission, the 100 attacker-owned Q&A records are part of the 5,040 benign-format training records, and the attacker adds 100 question-generation exchanges, yielding 5,140 records. Question-generation exchanges therefore make up about 1.9% of the training data (100/5,140), and all attacker-contributed records together make up about 3.9% (200/5,140).

For direct submission, the matched benign-SFT baseline includes the attacker-owned ordinary Q&A records. These are benign examples drawn from the same dataset as the other ordinary Q&A records. For hosted interactions, we simulate the attack through the Gemini API. The benign-SFT baseline uses only the benign training set and excludes these API exchanges, including the attacker-owned ordinary Q&A. We expect this difference to have negligible impact, since all attacker-contributed records are excluded from evaluation in both settings: extracted instructions are matched only against benign records the attacker never sees.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.