- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper discusses trigger-free data poisoning of abstractive summarization: an attacker rewrites the reference summaries of some fine-tuning examples so the model acquires a persistent behavior — inverted sentiment, injected toxicity, factual distortion, or demographic skew — that shows up on ordinary inputs with no inference-time trigger. Because the output space is open-ended and fluency and ROUGE survive, this is hard to spot.
The paper proposes two post-hoc defenses matched to two defender-access settings.
- Defense-1 assumes the fine-tuning corpus is available: rank examples by DataInf influence, take the top K% as candidates, filter them with four behavioral checks (sentiment disagreement, toxicity, factual inconsistency, representational disparity) against clean-calibrated thresholds, then gradient-ascent unlearn what survives, stopping when clean utility degrades past a tolerance.
- Defense-2 assumes only the checkpoint: probe it with character-, word-, sentence-, and document-level perturbations from prior summarization robustness work, measure how much more source content gets excluded from the summary under perturbation than without it, and call that Sensitivity to Adversarial Perturbations. A model is flagged when SAP exceeds a threshold calibrated from verified-clean reference checkpoints of the same architecture.
Evaluation spans 9 models (encoder-decoder and decoder-only), 6 summarization datasets, 4 poisoning objectives including 2 the paper introduces, and 7 contamination rates from 0.1% to 10%. There are also three defense-aware stress tests — mixed-influence poisoning, mixed-objective poisoning, and position-shifted probing — plus clean-model fragility controls. Headline numbers: 84.6% average behavioral recovery at 10% contamination against SISA's 76.1% and Antidote's 73.7%, and SAP at 0.93 AUROC against CLIBE's 0.84 and perplexity sensitivity's 0.71.
Critical strengths
The clean-model fragility controls are the right experiment and a lot of detection papers skip them. The obvious objection to "poisoned models are more perturbation-sensitive" is that any poorly-conditioned model is more perturbation-sensitive, so the signal might be detecting training pathology rather than poisoning. Section 6.2.3 and Figure 2 run undertrained, overfit, and reduced-data clean checkpoints through the same probes at the same architecture-specific thresholds and show the poisoned models separating from all three. That is what makes SAP a candidate poisoning signal rather than a fragility meter, and it is the reason I take the Defense-2 idea seriously even though I don't think the evaluation supports the numbers.
The breadth and the willingness to attack the assumptions are good. 9 architectures across 2 families, 6 datasets, 7 contamination rates, 2 new poisoning objectives, and 3 stress tests explicitly designed to weaken the signals the defenses depend on. The position-shifted probing result — AUROC falling from 0.94 on Lead-3 to 0.92 on Middle-3 and 0.89 on Tail-3 while mean SAP stays elevated — is the kind of self-directed attack that builds confidence, and Section 6.1.3's influence-region ablation (85.52% recovery from unlearning a high-influence poisoned subset against 58.2% random and 37.1% low-influence, with exact retraining as the 100% oracle) is a real mechanism analysis rather than another table of scores.
Critical weaknesses
My main problem is that every statistical claim about Defense-2 is computed against six negative examples. Appendix 9.2 states that for each architecture the clean-reference distribution comes from six verified-clean checkpoints, one per dataset, and that the threshold is τ_m = Q_0.95 of that distribution with "a target false-positive rate of 5%." An empirical 95th percentile of 6 observations sits between the 5-th and 6-th order statistics — it is essentially the maximum of 6 numbers, and it carries no 5% guarantee in any non-parametric sense.
- Everything downstream inherits this: the F1 and precision figures reported "at the architecture-specific clean-calibrated threshold," and the AUC, AUPR and TPR-at-5%-FPR numbers in Tables 3, 4 and 15, which Appendix 9.10.2 says are computed from clean and poisoned score distributions over the contamination sweep. With six negatives the standard error on an AUC is on the order of 0.1, which means Table 4's spread from 0.88 to 0.95 across nine models and six datasets is indistinguishable from noise, and TPR at 5% FPR is not estimable at all — 5% of six is 0.3, so the quantity reduces to "TPR at a threshold above the maximum clean score." No confidence intervals appear anywhere in the paper. The fix is cheap, since clean checkpoints are the easy thing to produce, but I don't see any of the Defense-2 operating-point numbers.
- Compounding this, those six references are obtained by "fine-tuning the architecture on each of the six clean summarization datasets using the same training configuration as the corresponding poisoned models." That is a stronger assumption than the model-only threat model admits. Defense-2 exists because the defender has a suspicious checkpoint and no fine-tuning data, yet calibration requires clean fine-tuning data for the same task & architecture & training recipe — and a way to know those references are clean, which is the very property under audit. If a defender can produce a verified-clean fine-tuned model on the target task, the auditing problem is much less pressing than the paper's framing suggests.
My second concern is that Defense-1's localization signal is the attacker's own selection criterion, and the natural adaptive attack against it is never run. Section 5 says the paper follows influence-targeted poisoning in which examples are drawn from a high-influence region, and Defense-1 then ranks by influence and keeps the top K%. Even under that favorable match the candidate set captures only 76–84% of the poisoned examples at 10% contamination.
- The mixed-influence stress test spreads the budget evenly across 1/3 of the influence ranking and recall falls to 73.8%, which is a useful check, but the obvious adaptive attack is to draw the poison entirely from the low-influence region, and that experiment is absent. Section 6.1.3 is what makes the omission conspicuous: unlearning a low-influence poisoned subset recovers only 37.1% of the behavior. If an attacker confines poisoning to low influence, the paper's own numbers suggest localization would find little and unlearning what it finds would undo little. The authors may argue "low-influence examples might be weak attack vehicles", but that is an empirical claim about attack success as a function of the influence percentile the poison is drawn from, and it is exactly the sweep the paper needs but lacks.
Questions for authors
Please recalibrate SAP with substantially more clean reference checkpoints — varying seed, data order, and hyper-parameters within each architecture-dataset cell, which is cheap — and report the measured false-positive rate on held-out clean models rather than a target. With a proper negative set, what happens to the AUC spread in Table 4 and to TPR at 5% FPR? Bootstrap intervals on those numbers would settle whether the cross-model and cross-dataset differences mean anything.
On Defense-1, could you run the attack with poison drawn from the low-influence region and report both attack success and defense recovery, alongside a sweep over the influence percentile the attacker samples from? That single figure would turn the mixed-influence stress test into a real adaptive-attack evaluation, and Section 6.1.3 suggests you have most of the machinery already.
Is the clean reference for a given target drawn from the same dataset as the target, or pooled across all six? SAP appears to vary by dataset, so pooling a single per-architecture threshold over six datasets inflates the reference spread in a way that should be characterized. And more fundamentally, how should a real defender obtain "verified-clean" references in the model-only setting? If the honest answer is that Defense-2 requires an organization that already trains its own clean models on the task, I'd rather see that stated plainly than left to the appendix.
3 comments
Response to rebuttal
Thanks for your new results and clarification. I decide to maintain my score.
Response to Reviewer fFqx
Thank you for reviewing our response and for your constructive feedback throughout. We have uploaded a revised manuscript that incorporates the changes discussed. Section 5 describes the SAP calibration with 108 verified-clean checkpoints per architecture and the measured held-out false-positive rate (4.73%). SAP detection is now reported separately at each contamination rate in Table 10 (Appendix 9.10.2). The clean-reference assumption is stated explicitly in Sections 3 and 4.2, and the low- and middle-influence poisoning results, including attack success and recovery, are reported in Section 6.1.5. We hope these revisions address your concerns.
Response to Reviewer fFqx
We thank the reviewer for the careful assessment and constructive suggestions. We appreciate the recognition of our clean-model fragility controls, broad experimental evaluation, and defense-aware stress tests. We address the concerns about statistical reliability, clean-reference calibration, and adaptive low-influence poisoning below.
Statistical Reliability and Clean-Reference Calibration:
Clean references: For each architecture-dataset pair, we trained clean checkpoints using 18 hyperparameter configurations, varying the learning rate (1e-5, 2e-5, 5e-5), batch size (8, 16), and number of training epochs (2, 3, 5), yielding 108 clean checkpoints per architecture (972 total), with matching poisoned checkpoints. SAP was computed individually for every checkpoint, the six values reported per architecture are dataset-level averages. For calibration, we use 54 clean checkpoints per architecture to estimate the empirical 95th-percentile threshold τₘ, reserving the remaining 54 for held-out evaluation. Across 486 held-out clean checkpoints, the observed FPR is 4.73% (23/486), compared with the nominal 5% target.
SAP calibration: Thresholds are set per architecture, not per dataset. Clean-reference SAP scores from all six datasets are pooled into a single distribution, and its 95th percentile defines τₘ, which is applied to targets from every dataset. Clean SAP varies only modestly across datasets. For Qwen, it ranges from 3.5 to 3.8, below the calibrated threshold τ_Qwen = 4.45 (Section-6.2.1), so pooling does not place any dataset’s clean models near the threshold. We will report per-dataset clean SAP distributions and false-positive rates for all architectures in the final version.
Statistical reliability: On Qwen-MultiNews (Table-3), SAP achieves AUROC 0.93 (95% CI: 0.80 - 0.98), AUPRC 0.91, F1 0.86, and TPR@5% FPR 0.81 (0.75 - 0.86). Across all nine architectures and six datasets, contamination-stratified AUROCs are 0.82 (0.68 - 0.91) at 0.1-0.25%, 0.94 (0.82 - 0.98) at 0.5 - 1%, and 0.99 (0.93 - 1.00) at 2.5 - 10%, confirming reduced separability at very low contamination. Most differences between architectures and datasets in Table-4 fall within overlapping confidence intervals. We will include checkpoint-level intervals and the held-out calibration protocol in the final version.
Obtaining verified-clean references: Defense-2 assumes the defender can fine-tune the same public base architecture on trusted public task data. Such references are clean by construction, and they do not require the suspicious model’s training data, recipe, or poisoning objective. They cannot reveal whether a third-party checkpoint is poisoned, which is the setting Defense-2 addresses. This fits organizations that deploy or audit third-party fine-tuned models but cannot inspect how those models were trained. We will state this assumption explicitly in Section-3.
Adaptive low-influence poisoning and influence sweep:
Defense-1 deliberately uses the same signal as the attack. Influence-based poisoning works because high-influence samples drive model behavior, so evading localization forces the attacker to give up that leverage.
Beyond the influence-region analysis in Section-6.1.3, we evaluated the adaptive attack in which all poisoned samples are drawn from the middle and low-influence region. At 10% contamination, this attack success falls from 81.2-90% under standard high-influence poisoning to 44-57% for middle-influence and only 17-19.3% for low-influence. Confining poison to low-influence samples therefore weakens the attack itself, so evading influence-based localization comes at the cost of attack effectiveness.
As Defense-1 measures influence on the poisoned model, poisoned samples tend to rise in the ranking. The Top-20% set captures 49% of middle-influence and 28% of low-influence poison. Defense-1 recovers 80.2% of the behavior under middle-influence poisoning, close to the standard setting (84.6%), and 52% under low-influence poisoning, where the attack induces only a small shift to begin with. Defense-1 therefore remains effective where the adaptive attack remains effective. The 37.1% in Section-6.1.3 is consistent with this result. It comes from unlearning a low-influence subset within the poisoned model, whose samples contribute little to the learned behavior. We will add these results to Section-6.1.3.