acceptodds
Under review as a conference paper at ICLR 2027

Do Defenses Against LLM Extraction Work Across Attacks? A Lifecycle Benchmark of Black-Box Model Extraction

Abstract

Large language models (LLMs) deployed through text-only APIs face model extraction risks, as adversaries can collect their responses to train surrogates that reproduce their capabilities. While prior work has developed diverse attacks and defenses, evaluations remain fragmented across access assumptions, model configurations, query budgets, and security objectives, limiting comparability across methods. To address this gap, we introduce a unified benchmark covering six extraction attacks, ten defenses, and two adaptive attacks that paraphrase or back-translate protected responses before surrogate training. The benchmark controls model configurations, query data, budgets, and held-out evaluation conditions within each comparison while preserving attack-specific querying and training procedures. We measure surrogate capability, fidelity to the victim, output quality using Rep-4, and query-budget sensitivity; defenses use their own security metrics paired with surrogate performance. For the adaptive attacks, we jointly measure provenance-detector scores and the capability and fidelity of surrogates trained on rewritten responses. The benchmark thus provides a reproducible basis for comparing extraction methods and their interactions with defenses under text-only access. Code and artifacts are available in our anonymous repository at https://anonymous.4open.science/r/mea-benchmark-20B1.

Primary area:
datasets and benchmarks
Keywords:
large language models, black-box model extraction, security benchmarking, model extraction defenses, adaptive attacks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VDBkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for black-box LLM extraction under text-only API access. It is organized around four "lifecycle" stages: query acquisition, victim interaction, surrogate training and evaluation.

Critical strengths
  1. A real problem and potentially useful infrastructure. Extraction attacks and defenses are currently evaluated under incompatible victims, budgets, data and metrics. The paper reimplements 18 methods with a shared query pool, shared victim transcripts, held-out evaluation and released run manifests. That engineering effort could serve as a starting point for a more relevant benchmark.

  2. A sound methodological stance on defenses. The authors do not collapse heterogeneous defenses into a single score. Each defense family gets its own objective-specific metric, paired with the utility of the resulting surrogate. Adaptive rewriting is evaluated after surrogate training rather than only on the rewritten text. Both choices address real weaknesses of prior evaluations.

Critical weaknesses

The threat model targets last-generation chat models, not the reasoning-model distillation that defines LLM extraction risk in 2026. Both victims are non-reasoning instruct models (Llama-3.3-70B-Instruct, Qwen2.5-72B-Instruct). Generation is greedy with at most 1536 new tokens, so long chain-of-thought traces cannot fit. Today, extraction in practice means distilling reasoning traces from frontier reasoning models.

As a result, it is unclear whether any headline finding transfers to the setting that practitioners and defenders currently care about. A benchmark paper's contribution is to serve as a reference point, so this limits significance more than any single implementation issue.

Questions for authors
  1. Have you run any part of the benchmark with a reasoning victim (an open-weight model emitting long CoT), and with CoT-hidden or CoT-summarized API variants? Do the anti-distillation and provenance conclusions hold there?

  2. Trace Rewriting and Antidistillation Sampling were proposed for reasoning traces. What justifies evaluating them on short chat responses?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VDBk

Thank you for raising the question of whether our conclusions extend to reasoning models.

W1 / Q1. Extraction from a reasoning victim

Answer

We evaluated six response defenses against SeqKD using QwQ-32B as the victim and Qwen2.5-7B as the initial surrogate. We trained on full visible responses to the same 1000 queries (B=1000B=1000) in each condition. We increased the generation limit from 1536 to 8192 tokens for QwQ-32B, and 26 of the 1000 clean responses reached the new limit.

Evidence

Anti-distillation. Clean SeqKD raises macro-average accuracy from 58.04% to 58.95%. ADS yields the largest accuracy drop relative to clean (0.45 percentage points) and reduces final-answer BERTScore F1 from 0.0395 to 0.0051. Under ADS, 915 of the 1000 victim training responses reach the generation limit, and none contain a final answer. DOGe and Trace Rewriting each lower accuracy by at most 0.16 points and BERTScore F1 by at most 0.0022. The clean surrogate emits no explicit reasoning block on the evaluation prompts, so these comparisons measure changes in task performance and final-answer fidelity.

Provenance. We probe each protected surrogate and its matched clean control with the same 1000 extraction queries.

Detector (score)ProtectedMatched clean
ADFP (−log⁡10(p)-\log_{10}(p))0.00830.00830.00570.0057
GINSEW (PsnrP_{\mathrm{snr}})1.156×10−181.156 \times 10^{-18}2.739×10−212.739 \times 10^{-21}
Radioactivity (pp-value)0.1180.1180.2040.204

ADFP stays close to zero and Radioactivity yields a modest decrease in pp-value. GINSEW shows the largest relative change on its own scale, with a protected score about 420 times its clean score.

Manuscript changes

We placed the QwQ response-defense evaluation in Appendix E.1, which reports the evaluation setup, full surrogate results, and paired provenance scores.

Q2. Why evaluate ADS and Trace Rewriting on chat responses?

Answer

ADS adjusts the next-token distribution during decoding, and our Trace Rewriting implementation rewrites the full response (Appendix B.3). Both can operate on ordinary chat responses without an explicit reasoning trace. Such responses are relevant to extraction because an API may expose only the final answer.

Evidence

In our ordinary chat-response experiments, ADS lowers surrogate accuracy by 3.51 percentage points for QEDKS and 3.07 for LoRD, and lowers BERTScore F1 for both. Trace Rewriting lowers accuracy for SeqKD, QEDKS, and SODA and raises it for Model Leeching, LoRD, and GAD (Fig. 2 and Table 13). These results show that the effect of each defense on surrogate performance varies across attacks in this setting. In the QwQ-32B experiment with visible reasoning traces, both defenses change SeqKD accuracy by less than half a point. ADS produces the largest drop in final-answer fidelity, and Trace Rewriting shortens the average training response from 1891 to 1365 tokens with little change in fidelity (Appendix E.1).

Manuscript changes

We have clarified in §3.2 that the ordinary chat-response experiments test whether ADS and Trace Rewriting also suppress extraction in that setting. Appendix E.1 reports the results for both defenses in the visible-trace setting.

Thank you for your detailed response. The additional QwQ experiments partially address my concern, and I appreciate the effort.

My main remaining concern is generalization to frontier models on agentic tasks. Evaluating the extraction method with frontier models such as GLM-5.3 and DeepSeek-v4-flash on agentic benchmarks such as Terminal-Bench would show whether the approach holds beyond the current model family and in realistic, multi-step settings. I think this evidence would substantially strengthen the paper and would be the key factor in raising my assessment.

I understand that such experiments are resource-intensive and may not be feasible within the revision period. Even so, I cannot raise my score based on the current version. I would be glad to reconsider if the authors can provide these results, or at least a limited-scale pilot on a subset of Terminal-Bench tasks, with a clear discussion of the remaining limitations.

AuthorsOpenReview ↗

Follow-up to Reviewer VDBk

Thank you for your follow-up and for suggesting a limited-scale pilot on Terminal-Bench. We have completed an additional experiment with DeepSeek-V4-Flash as the victim and report the results below.

Extraction from a frontier victim on agentic tasks

Answer

We evaluated SeqKD, SODA, and LoRD with DeepSeek-V4-Flash-0731 as the victim and Qwen3.5-9B as the initial surrogate on 20 fixed Terminal-Bench 2.1 tasks. This extends the evaluation to a new victim–surrogate pair and multi-step terminal interaction. All three attacks reuse victim trajectories from 96 independent synthetic training exercises, covering file processing, code repair, algorithm implementation, and numerical tasks. The victim solved all 96 exercises, yielding 493 assistant turns. SODA starts from the SeqKD checkpoint, while LoRD starts from the initial surrogate.

Evidence

All five models use Terminus 2 through Harbor, with the same evaluation tasks and limits. Each task receives one attempt, with temperature 0, a 16384-token context, up to 20 interaction rounds, and at most 2048 generated tokens per round. Success is determined by each task's verifier.

ModelTasks passedSuccess rateSuccess-set overlap with victim
Victim (DeepSeek-V4-Flash)7/2035%—
Initial surrogate (Qwen3.5-9B)4/2020%4/7
SeqKD1/205%1/7
SODA3/2015%3/7
LoRD0/200%0/7

Success-set overlap is the Jaccard similarity between the model's and victim's sets of passed tasks. Here, every task passed by a surrogate was also passed by the victim, so the union contains seven tasks.

Task performance and agreement. None of the attacks improves aggregate success or success-set overlap over the initial surrogate. SeqKD loses three previously passed tasks and gains none. SODA newly solves cobol-modernization, which the victim solves and the initial surrogate fails, but loses two previously passed tasks. LoRD training causes output degeneration into repeated newlines, preventing valid agent actions. Its 20 tasks end in 15 timeouts and five context-limit failures.

These results show that training on successful victim trajectories does not ensure improved multi-step task completion under this setup. SODA transfers one additional successful outcome while also losing existing successes.

Limitations. This pilot covers one model pair and 20 tasks with one attempt each. The victim and initial surrogate exhaust the context on 11 and 14 tasks, respectively, so the results reflect the specified interaction limits. The training exercises are independently constructed and simpler than the evaluation tasks. The experiment provides evidence about extraction under this configuration; broader task coverage, repeated attempts, and larger interaction budgets would be needed to assess generality.

Manuscript changes

We added Appendix E.2 with the experimental setup, aggregate and per-task results, success-set overlap, and limitations (Tables 10 and 11).

Reviewer_9UtaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for text only LLM extraction. It has four lifecycle stages: query acquisition, victim interaction, surrogate training, and evaluation. It covers six attacks (SeqKD, Model Leeching, LoRD, SODA, GAD, QEDKS). The attacks are evaluated on ten defenses grouped by defensive objective: query detection, anti-distillation, provenance, and lineage verification. From the experimental result, the paper shows that defenses work against some attacks but not all of them. It also suggests that extraction attack rankings depend on the evaluation metric. Anti-distillation effectiveness also varies across attacks. Besides, increasing the query budget doesn't always help.

Critical strengths
  1. The paper provides a structured framework for diverse attacks and defense mechanisms in LLM extraction. It evaluates six model extraction attacks, ten defenses, and two adaptive attacks.
  2. The proposed framework provides several valuable insights into LLM model extraction and defense effectiveness. The results show that no single extraction attack consistently performs best across different evaluation metrics. Besides, existing defenses also don't consistently perform across different attacks. These findings provide a better understanding of the limitations of existing defenses and highlight the importance of more robust protection mechanisms.
Critical weaknesses
  1. The paper introduces six extraction attacks but in the defense evaluation all the defenses are evaluated against only 3 attacks (SeqKD, QEDKS, and SODA). Evaluating defenses against the remaining attacks would provide stronger evidence about the true effectiveness of all the defense mechanisms.

  2. The paper uses different victim surrogate model configurations for attack (Llama-3.3-70B, Llama-3.1-8B) and defense(Qwen2.5-72B-Instruct, Qwen2.5-7B) evaluations. The intuition of using different family models is not present in the paper. Evaluating the main attack defense combinations across multiple model families would strengthen the generalizability of the findings.

  3. The paper shows performance comparisons without doing repeated experiments across random seeds. As performance of different attacks are very small, showing standard deviations or confidence intervals would help to strengthen the findings.

  4. The paper also doesn't provide the performance of the initial surrogate checkpoints or victim models on the same evaluation benchmarks. For example, SeqKD achieves nearly identical average accuracy in different query budgets (60.70% at 100 queries, 61.10% at 1,000 queries, and 60.54% at 10,000 queries). But it doesn't show the improvements from the initial surrogate performance. Including the result of both initial surrogate and victim baselines would help to better interpret the results.

Questions for authors
  1. Can the paper add a full attack defense matrix across six extraction attacks instead of putting the results of three attacks?
  2. What is the result of initial surrogate and victim scores on all capability and fidelity metrics?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9Uta

Thank you for the helpful suggestions on defense coverage, cross-family evaluation, seeds, and baselines.

W1 / Q1. Full attack–defense matrix

Answer

We completed the six-attack response-defense matrix by running all six response defenses against Model Leeching, LoRD, and GAD. We added 18 defended runs and three matched clean controls on the original Qwen pair at B=1000B=1000, with one seed per condition.

Evidence

ADS lowers LoRD accuracy by 3.07 points and BERTScore F1 by 0.128. For Model Leeching, 13/1000 ADS-protected responses pass the parser and accuracy rises by 1.03 points relative to clean. Among the added attacks, GINSEW rises for Model Leeching and GAD and falls for LoRD. ADFP scores never rise, and Radioactivity rises only for GAD.

Manuscript changes

The expanded results appear in Section 4.2, Figs. 2 and 3, and Tables 9 to 11.

W2. Choice of model families and cross-family evaluation

Answer

Model availability on our cluster changed during the project. We ran the main attack evaluation on Llama and retained the Qwen defense experiments. Each defense comparison uses a clean surrogate trained with the same attack on the same model pair. We compare provenance on Qwen and Llama for SeqKD and QEDKS.

Evidence

Provenance. We compare the Qwen defense results (Table 11) with the clean and defense-only controls from the Llama rewriting experiments (Table 14) at B=1000B = 1000. Protected / matched clean score; higher values indicate stronger evidence.

DetectorSeqKD (Qwen)SeqKD (Llama)QEDKS (Qwen)QEDKS (Llama)
ADFP (−log⁡10(p)-\log_{10}(p))0 / 00 / 00.054 / 0.0370 / 0
GINSEW (PsnrP_{\mathrm{snr}})2.04×10−202.04 \times 10^{-20} / 1.40×10−201.40 \times 10^{-20}4.27×10−224.27 \times 10^{-22} / 5.10×10−215.10 \times 10^{-21}1.49×10−201.49 \times 10^{-20} / 6.33×10−226.33 \times 10^{-22}9.77×10−219.77 \times 10^{-21} / 5.01×10−205.01 \times 10^{-20}
Radioactivity (−log⁡10(p)-\log_{10}(p))0.49 / 0.450.51 / 0.057.60 / 7.582.95 / 2.26

Neither model pair shows consistent clean/protected separation. GINSEW rises under protection on Qwen and falls on Llama for both attacks. Radioactivity scores rise in all four comparisons, but both Qwen QEDKS pp-values are about 2.6×10−82.6 \times 10^{-8}. Training configurations differ across pairs, so these comparisons are descriptive.

Manuscript changes

We have clarified in §3.3 that each Qwen response-defense experiment runs the same attack on the same Qwen model pair in both the clean and defended conditions.

W3. Variation across random seeds

Answer

We evaluated all six attacks with three training seeds on the Llama pair at B=1000B = 1000. The table reports the mean ±\pm sample standard deviation.

Evidence

AttackACC (%)BERTScore F1Rep-4
SeqKD61.08±0.0961.08 \pm 0.090.337±0.0010.337 \pm 0.0010.124±0.00010.124 \pm 0.0001
Model Leeching61.11±0.5361.11 \pm 0.530.062±0.0220.062 \pm 0.0220.055±0.0040.055 \pm 0.004
LoRD57.85±0.3957.85 \pm 0.390.302±0.0120.302 \pm 0.0120.172±0.0180.172 \pm 0.018
SODA61.08±0.0461.08 \pm 0.040.337±0.0010.337 \pm 0.0010.123±0.0010.123 \pm 0.001
GAD55.11±0.9555.11 \pm 0.950.161±0.0460.161 \pm 0.0460.481±0.0890.481 \pm 0.089
QEDKS59.07±0.0759.07 \pm 0.070.350±0.0020.350 \pm 0.0020.118±0.0010.118 \pm 0.001

SeqKD, Model Leeching, and SODA differ in mean accuracy by at most 0.03 points, within their run-to-run standard deviations. Model Leeching has much lower fidelity, and QEDKS has the highest fidelity.

Manuscript changes

Table 1 now reports three-seed means and sample standard deviations, including per-task accuracy, and §4.1 discusses the variation across runs. §4.4 and Appendix E.1 identify the original training seed used for the budget comparison.

W4 / Q2. Initial-surrogate and victim baselines

Answer

Each victim's responses serve as the BERTScore reference for its model pair.

Evidence

Model pairCheckpointACC (%)BERTScore F1
LlamaLlama-3.1-8B-Instruct (initial surrogate)60.950.343
LlamaLlama-3.3-70B-Instruct (victim)64.62reference
QwenQwen2.5-7B (initial surrogate)58.420.044
QwenQwen2.5-72B-Instruct (victim)66.28reference

Llama SeqKD stays close to its initial 60.95% across budgets (60.70%, 61.10%, and 60.54%). The Llama and Qwen initial surrogates start 3.67 and 7.86 points below their respective victims. QEDKS improves accuracy by 4.92 points over the Qwen baseline.

Manuscript changes

Table 1 now reports the Llama initial-surrogate and victim baselines, including per-task accuracy. §4.4 compares SeqKD's accuracy across query budgets with the initial surrogate. The appendix reports per-task and macro-average accuracy for both Qwen reference checkpoints.

The rebuttal substantially improves the empirical support of the paper, and the remaining issues can mostly be fixed by clearer scoping and wording. Based on the new experiments, I am raising my score.

Thank you for reviewing the additional experiments. Your suggestions on defense coverage, cross-family comparisons, repeated runs, and model baselines helped us improve the evaluation.

Reviewer_DK5SOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper introduces a lifecycle benchmark for black-box LLM extraction, covering six extraction attacks, ten defenses, and two response-rewriting attacks. It evaluates task accuracy, behavioral fidelity, repetition, and defense-specific outcomes under matched conditions within each comparison. The results show that attack rankings depend on the metric, defense effectiveness varies across extraction procedures, and larger query budgets do not consistently improve extraction.

Critical strengths
  • The paper addresses a useful evaluation problem. Differences in models, query budgets, and training settings make results from separate extraction studies difficult to compare. Bringing existing methods into a shared evaluation framework has practical value.

  • The evaluation distinguishes different defense objectives. Capability suppression, provenance detection, and query detection are assessed separately. Evaluating surrogates trained on rewritten responses also helps connect response-level changes to their downstream effects.

Critical weaknesses
  • The contribution beyond consolidating existing evaluations is not sufficiently clear. The lifecycle largely follows the standard extraction pipeline, while the data sources and evaluation metrics are established. These choices are reasonable, but the paper needs to explain more clearly what substantive advance the protocol provides over existing evaluation practices. At present, the lifecycle seems more useful as an organizational framework than as a distinct methodological contribution.

  • The cross-attack evaluation is narrower than the claimed benchmark coverage suggests. Although six attacks and ten defenses are included overall, response-based defenses are evaluated only against SeqKD, QEDKS, and SODA; query detectors cover two attacks; and adaptive rewriting uses only SeqKD. In particular, Model Leeching, LoRD, and GAD are absent from the response-defense comparisons. Since fair comparison across attacks is the main contribution, these omissions limit how convincingly the experiments establish defense transferability.

Questions for authors
  • Which specific elements of the evaluation protocol go beyond existing practices? What comparisons or conclusions become possible because of these elements, rather than simply bringing existing implementations together?

  • Why were Model Leeching, LoRD, and GAD excluded from the response-defense comparisons? Are these combinations incompatible with the protocol, or were they omitted for practical reasons?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DK5S

Thank you for the careful review. Below we clarify what the protocol contributes and report the new coverage experiments.

W1 / Q1. What does the protocol add beyond existing practice?

Answer

The protocol uses matched controls to compare defense effects across attacks and measure what changes after adaptive rewriting.

Evidence

  1. Cross-attack comparison. Each clean/protected pair shares the budget and training configuration. Fixed-pool attacks also share ordered queries. ADS lowers QEDKS accuracy by 3.51 points and raises SODA accuracy by 0.57 points (Fig. 2).
  2. Two controls for rewriting. The clean surrogate establishes the baseline for evidence of mark inheritance. A drop after rewriting can indicate that an inherited mark was weakened only if the protected surrogate first scored above the clean baseline. The defense-only surrogate measures the change after rewriting. For SeqKD, back-translation lowers accuracy by 5.2 to 11.4 points relative to defense-only (Table 2).

Manuscript changes

The introduction now explains both controls and the comparisons they support.

W2 / Q2. Coverage of the defense and adaptive evaluations

Answer

The protocol supports the added combinations. We initially limited coverage because of computational cost.

Evidence

Response defenses. We evaluated all six response defenses against Model Leeching, LoRD, and GAD. This adds 18 defended conditions and one matched clean control per attack. As in the original defense experiments, we use the Qwen2.5-72B-Instruct → Qwen2.5-7B pair, B=1000B = 1000, and one seed per condition. Figs. 2 and 3 now show results for all six attacks.

Anti-distillation. ΔACC\Delta\mathrm{ACC} (percentage points) / ΔBERTScore F1\Delta\text{BERTScore F1}, both defended minus clean.

DefenseModel LeechingLoRDGAD
ADS+1.03+1.03 / +0.102+0.102−3.07-3.07 / −0.128-0.128+0.79+0.79 / −0.011-0.011
DOGe−1.17-1.17 / +0.005+0.005−0.04-0.04 / +0.006+0.006−0.06-0.06 / −0.003-0.003
Trace Rewriting+0.48+0.48 / +0.003+0.003+0.44+0.44 / −0.062-0.062+1.32+1.32 / −0.015-0.015
  • ADS reduces both accuracy and fidelity for LoRD, but not for the other two attacks. This is consistent with the attack dependence in Fig. 2.
  • Under clean training, neither Model Leeching nor GAD improved macro-average accuracy over its initial surrogate. Small defended-minus-clean changes in accuracy therefore tell us little about whether these defenses suppress gains from extraction.
  • For Model Leeching, ADS changes the response format. Only 13 of the 1000 protected responses pass the attack's parser, and the resulting surrogate has higher accuracy than its matched clean counterpart.

Provenance. Protected / matched clean score. For ADFP and Radioactivity we report −log⁡10(p)-\log_{10}(p); for all three detectors, higher values indicate stronger evidence of inheritance.

DetectorModel LeechingLoRDGAD
ADFP (−log⁡10(p)-\log_{10}(p))0.21 / 0.270.01 / 0.100.00 / 0.00
GINSEW (PsnrP_{\mathrm{snr}})2.97×10−202.97 \times 10^{-20} / 7.83×10−217.83 \times 10^{-21}2.35×10−202.35 \times 10^{-20} / 7.67×10−207.67 \times 10^{-20}2.60×10−192.60 \times 10^{-19} / 6.54×10−216.54 \times 10^{-21}
Radioactivity (−log⁡10(p)-\log_{10}(p))1.73 / 2.110.94 / 1.960.81 / 0.61

For ADFP and Radioactivity, the protected surrogate scores above its clean control in only one of six cases (Radioactivity on GAD). GINSEW scores higher under protection for Model Leeching and GAD but lower for LoRD. Across attacks, the detector scores show no consistent separation between protected and matched clean surrogates (§4.2).

Query detection. On Model Leeching, MMD and SEAT each flag 20/20 attack batches and 0/4 held-out benign batches. PRADA flags 1/20 and 1/4. MMD and SEAT also flag all evaluated QEDKS attack batches and rarely flag SeqKD's fixed-pool batches. LoRD, SODA, and GAD use the same fixed-pool queries as SeqKD.

Adaptive rewriting. Both rewrites lower BERTScore F1 under all three defenses for SeqKD and QEDKS. Maximum accuracy losses relative to defense-only are 11.4 and 1.98 points, respectively. ADFP scores stay at zero. Both rewrites move Radioactivity toward the clean score for SeqKD and lower its detector pp-value for QEDKS.

Manuscript changes

We have updated §4.2, Figs. 2 and 3, and Tables 9–11 with the six-attack response-defense results. We added Model Leeching to the query-detection analysis in §4.2 and to Table 12. We added QEDKS to the adaptive analysis in §4.3, Table 2, Fig. 4, and Table 14.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Dear Reviewer DK5S,

Thank you again for your detailed feedback. We wanted to briefly follow up on the revisions addressing the protocol contribution and the scope of the evaluation.

  • Evaluation protocol (W1 / Q1). The Introduction now explains what the matched controls let us measure. Matched clean and protected runs allow us to compare how a defense affects different extraction attacks. In the rewriting experiments, the clean surrogate establishes whether protection produced a measurable provenance signal, and the defense-only surrogate provides the reference for measuring the effect of rewriting.
  • Response-defense coverage (W2 / Q2). We evaluated all six response defenses against Model Leeching, LoRD, and GAD, adding 18 defended conditions and three matched clean controls. The revised manuscript now reports all six response defenses across all six attacks. For example, ADS reduces LoRD accuracy by 3.07 percentage points and BERTScore F1 by 0.128, while accuracy increases slightly for the other two added attacks (§4.2, Figs. 2–3, Tables 9–11).
  • Query detection (W2). We added Model Leeching to the query-detection evaluation. MMD and SEAT each flag all 20 evaluated Model Leeching attack batches and none of the four held-out benign batches (§4.2, Table 12).
  • Adaptive rewriting (W2). We extended the DIPPER and back-translation experiments to QEDKS. Both rewrites move the Radioactivity score toward the clean baseline for SeqKD, while both lower the detector’s p-value further for QEDKS (§4.3, Fig. 4, Tables 2 and 14).

We hope these revisions address your concerns about what the protocol adds and how broadly the findings are supported. If so, we would appreciate your reconsidering your assessment in light of the revised manuscript. We would be happy to clarify any remaining questions.

Thanks for the replies. The authors have answered most of my questions, especially by conducting additional experiments for a complete cross-attack evaluation. Therefore, based on how this rebuttal answers my questions, I would raise the score to 3. However, the novelty issues of its matched control and lifecycle still exist. In fact, I think this paper might be a better fit for a Systematization of Knowledge (SoK) paper on security venues (such as IEEE S&P) than a benchmark paper, as its main contribution seems to be a general platform for comparing existing work that motivates new findings for the community, rather than proposing new evaluation gaps, datasets or metrics.

Thank you for reviewing our responses and the additional experiments. Your suggestions helped us complete the cross-attack evaluation and clarify the scope of the paper’s contributions. We appreciate your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.