- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on the privacy perspective of tokenizers of large language models. The authors first propose a membership inference attack against tokenizers that outperforms an existing baseline. Based on the MIAs, the authors then propose a differentially private tokenizer that can defend against the MIAs.
Critical strengths
- The paper investigates a practical threat model as the tokenizer is usually a public component of LLMs.
- The performance gain of the proposed MIA is significant compared to the baseline.
Critical weaknesses
- The proposed MIA requires a large amount of in-distribution data to train shadow tokenizers. This assumption is not practical in real-world scenarios.
- The tokenizers are evaluated only on bytes per token. However, tokenization is mostly a step for LLMs. The impact of tokenization on downstream tasks (e.g., language modeling) is not evaluated.
- Only one dataset is used for experiments.
- Some designs of the proposed differentially private tokenizer are not well justified. It is unclear why the authors picked those certain designs. For example, the binary word presence vector can be compared with a vector of clipped word counts, as a word can appear many times in a document. The uniform subsampling can be compared with frequency-aware sampling.
- Missing analysis on and .
Questions for authors
See weaknesses
5 comments
W3: Only one dataset is used for experiments.
To address the Reviewer’s comment, we repeated the undefended and DP-BPE experiments at a vocabulary size of 200,000 on the FineWeb dataset, which differs from C4 in crawl (and hence, websites included) and filtering. Overall, we observe that the trends on FineWeb are very similar to those on C4.
We built the pool of 19,628 websites as for C4 and kept the rest of the setup.
| Undefended | DP-BPE, | DP-BPE, | DP-BPE, | |
|---|---|---|---|---|
| Bytes per token | 5.04 0.00 | 4.00 0.00 | 4.51 0.00 | 4.88 0.00 |
| Compression Rate | 1.2 0.1 | 1.0 0.0 | 1.1 0.0 | 1.0 0.0 |
| Token-Count Attack (ours) | 63.8 0.5 | 1.0 0.0 | 1.1 0.0 | 1.2 0.1 |
| Vocabulary Overlap | 16.9 0.5 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 |
| Token-Identity Attack (ours) | 53.8 0.2 | 1.0 0.0 | 1.0 0.0 | 0.9 0.0 |
| Frequency | 10.1 0.3 | 1.0 0.0 | 1.0 0.1 | 1.0 0.0 |
| Naive Bayes | 10.0 0.0 | 1.1 0.1 | 1.1 0.1 | 1.1 0.1 |
| Merge-Order with Bernoulli (ours) | 56.1 0.2 | 1.0 0.0 | 0.9 0.0 | 0.6 0.0 |
| Merge-Order with Bradley-Terry (ours) | 47.6 0.3 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 |
As on C4, our MIAs outperform the prior attacks, and every undefended attack TPR is within 3 points of its value on C4. The strongest prior attack, Vocabulary Overlap, reaches 16.9% TPR@1%FPR against the undefended tokenizer. DP-BPE reduces every attack to chance and keeps 79%, 89% and 97% of the bytes per token of the undefended tokenizer at , 3, and 10. We added the new Table 3 to Section 6.3 and a new Appendix G.9, named FineWeb in Section 6.1, and summarized it in Section 6.2.
W4: Some designs of the proposed differentially private tokenizer are not well justified. It is unclear why the authors picked those certain designs. For example, the binary word presence vector can be compared with a vector of clipped word counts, as a word can appear many times in a document. The uniform subsampling can be compared with frequency-aware sampling.
We considered several alternatives when designing DP-BPE and chose word presence with uniform subsampling because this combination gives the highest bytes per token at the same privacy guarantee. A word is released only if its noisy count exceeds the threshold , and grows with the largest amount one website can add to the count of a word. Word presence limits this amount to 1, so is lowest and more words are released. Uniform subsampling matches this choice, since it also ignores how often a word occurs within a website. Following the Reviewer's suggestion, we explicitly compared different variants and design choices at and a vocabulary of 200,000. We compare the subsampling rules at , where the cap applies to many websites.
| Variant | Bytes per token | Token-Count | Token-Identity | Merge-Order, Bernoulli | Merge-Order, Bradley-Terry |
|---|---|---|---|---|---|
| Word presence, uniform subsampling (paper) | 4.72 0.01 | 1.2 0.1 | 1.1 0.0 | 0.4 0.0 | 1.0 0.0 |
| Counts clipped at 2 | 4.68 0.01 | 1.2 0.1 | 1.0 0.0 | 0.4 0.0 | 1.0 0.0 |
| Counts clipped at 4 | 4.60 0.01 | 1.3 0.0 | 1.1 0.1 | 0.4 0.0 | 1.0 0.0 |
| Counts clipped at 16 | 4.36 0.01 | 1.2 0.0 | 1.0 0.0 | 0.4 0.0 | 1.0 0.0 |
| , uniform subsampling | 4.71 0.01 | 1.1 0.0 | 1.0 0.0 | 0.4 0.0 | 0.9 0.0 |
| , most frequent words kept | 4.70 0.01 | 1.2 0.0 | 1.0 0.0 | 0.5 0.0 | 0.9 0.0 |
| , sampled proportionally to frequency | 4.71 0.01 | 1.2 0.1 | 1.0 0.0 | 0.4 0.0 | 1.0 0.0 |
Clipped counts lower bytes per token from 4.72 to between 4.68 and 4.36. This trend can be explained as follows: A website adds up to instead of 1 to the count of a word when counts are clipped at . Its vector then has a larger norm, so more websites are rescaled to . The rescaling lowers the count that each website adds to a word, while the noise stays the same and the release threshold rises by . Fewer words are therefore released.
The two frequency-aware rules give 4.70 and 4.71 vs. 4.71 for uniform subsampling. Frequency-aware sampling therefore gives no gain, and we keep the simpler rule.
Every variant leaves the attacks at chance. We added the new Table 20 to Appendix I.2 and summarized it in Section 6.3.
W5: Missing analysis on and .
Conceptually, sets the noise scale , and the release threshold depends on both and , so the guarantee holds for every value of either. To address the Reviewer's comment, we ran additional experiments, where we ablate over these parameters at and a vocabulary of 200,000, keeping the other values fixed.
| Bytes per token | Token-Count | Token-Identity | Merge-Order, Bernoulli | Merge-Order, Bradley-Terry | ||
|---|---|---|---|---|---|---|
| 30 (paper) | 50,000 (paper) | 4.72 0.01 | 1.2 0.1 | 1.1 0.0 | 0.4 0.0 | 1.0 0.0 |
| 3.75 | 50,000 | 4.70 0.01 | 1.2 0.0 | 1.0 0.1 | 0.4 0.0 | 1.1 0.1 |
| 7.5 | 50,000 | 4.71 0.01 | 1.3 0.0 | 1.1 0.1 | 0.4 0.0 | 1.0 0.0 |
| 15 | 50,000 | 4.72 0.01 | 1.1 0.0 | 0.9 0.0 | 0.4 0.0 | 1.0 0.0 |
| 60 | 50,000 | 4.70 0.01 | 1.2 0.0 | 1.0 0.0 | 0.5 0.0 | 1.0 0.0 |
| 120 | 50,000 | 4.52 0.01 | 1.1 0.1 | 1.0 0.1 | 0.5 0.0 | 1.0 0.0 |
| 30 | 1,000 | 4.48 0.01 | 1.1 0.0 | 1.1 0.1 | 0.7 0.0 | 0.9 0.0 |
| 30 | 2,500 | 4.64 0.01 | 1.2 0.1 | 0.9 0.0 | 0.5 0.0 | 0.9 0.0 |
| 30 | 5,000 | 4.71 0.01 | 1.1 0.0 | 1.0 0.0 | 0.4 0.0 | 0.9 0.0 |
| 30 | 10,000 | 4.73 0.01 | 1.1 0.0 | 1.0 0.1 | 0.4 0.0 | 1.0 0.0 |
| 30 | 25,000 | 4.73 0.01 | 1.1 0.0 | 1.0 0.0 | 0.4 0.0 | 1.0 0.0 |
| 30 | 100,000 | 4.72 0.01 | 1.1 0.1 | 1.0 0.0 | 0.4 0.0 | 1.0 0.0 |
The attacks remain at random for all values. At , the utility of DP-BPE is robust to the choice of these hyperparameters over a wide range, so they do not need to be tuned on the private corpus. For from 3.75 to 60 and from 5,000 to 100,000, bytes per token stays between 4.70 and 4.73, against 4.72 in the paper. At , bytes per token drops to 4.52, because the noise grows with while the contribution of almost every website to the word counts stays the same. At , it drops to 4.48, because the cap removes words from most websites. We added the new Table 19 to Appendix I.2 and summarized it in Section 6.3.
If the above answers address the Reviewer's concerns, we would greatly appreciate it if they could consider updating their score. We are also happy to address any remaining questions.
We thank the Reviewer for recognizing our practical threat model and the significant gain of our MIA over prior attacks. We address all comments below.
Since tokenizers train on web text and large web crawls such as Common Crawl are public, an adversary can download other websites from the same source. In-distribution shadow data is the standard MIA assumption, as in LiRA. The shadow data must come from the same distribution as the training data, but it does not have to be the training data itself.
To further address the Reviewer's comment, we ran additional experiments where the adversary does not hold the tested websites but other websites instead, matching the availability of web crawls like Common Crawl. We trained the shadow tokenizers on one half of the C4 websites of the paper and tested the other half, so no shadow tokenizer is trained on a tested website. For comparison, the first two rows use the shadow tokenizers of the paper. In the first row, as in Table 1, half of them are trained with the tested website. The second row uses only those trained without it, as in the new setting. The table reports TPR@1%FPR of the Token-Count Attack at a vocabulary of 200,000.
While performance is smaller than when the shadow tokenizers are trained on the tested websites, the attack still detects 28.5% of the member websites, against 1% for random guessing. We added a new Appendix G.8 with the new Table 14, summarized it in Section 6.4, and added a sentence to Section 3.
To address the Reviewer's comment, we pretrained GPT-2-small on C4 with the undefended tokenizer and with DP-BPE at ε∈{1,3,10}. We trained every model for 3,094 steps on 1.62×109 tokens with three seeds and evaluated it on downstream tasks.
The first table below reports zero-shot accuracy on three benchmarks. The second table reports accuracy after fine-tuning on two GLUE tasks.
DP-BPE does not lead to a significant drop in downstream performance. After fine-tuning, it is within 0.1 points of the undefended tokenizer on QNLI at every ε. On MRPC it is 2.1 and 1.6 points lower at ε = 1 and 3 and 1.0 point higher at ε=10. On BLiMP, PIQA and ARC-Easy it is 1.5 to 5.5 points lower, and the gap shrinks as ε grows, as bytes per token does. We added the new Table 4, described the evaluation in Section 6.1, summarized the results in Section 6.3, and added a new Appendix I.4 with the training and evaluation details.