- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper discusses the vulnerabilities of dataset distillation to malicious perturbation and proposes NOVA to address this challenge. NOVA first jointly aligns clean and perturbed synthetic datasets to the same real-data target and ensures the matching holds across a neighborhood of the delivered set rather than at a single point. Additionally, this paper gives theoretical analysis along with experimental verification to demonstrate that NOVA improves the performance of models trained on a maliciously modified condensed dataset.
Critical strengths
-
The paper discusses an important problem of defending against malicious modification of the distilled dataset.
-
The proposed NOVA provides a straightforward framework to mitigate the perturbation by accommodating several matching objectives without changing the downstream targets.
Critical weaknesses
-
The main concern is the scale of the evaluated dataset. Dataset distillation mainly works on large-scale datasets of high-resolution images to reduce training cost. The datasets discussed here are primarily based on small, low-resolution benchmarks. This raises my doubts about the practical significance of NOVA. It is recommended to evaluate NOVA on at least one or two scalable backends on a substantially larger benchmark to validate its practical significance.
-
The objective of DD perturbation is quite similar to the definition of unlearnable examples. Lots of defense methods have been proposed to mitigate or purify the noise caused by unlearnable examples, thus enhancing the performance of the trained model. The author should clearly state the main difference of its defense and existing defense methods and, at a minimum, compare NOVA in the experiments with existing noising defense and mitigation methods.
-
The author should discuss the robustness or performance of NOVA against different perturbation budgets. It is recommended to provide accuracy-versus-attack-budget curves for representative attacks and backends.
Questions for authors
-
Can the authors evaluate NOVA using one or two scalable DD backends on a larger, higher-resolution benchmark to verify whether its robustness gains remain practically meaningful?
-
What distinguishes NOVA from defenses against unlearnable examples, and how does it compare with representative mitigation or purification methods in terms of both clean-set utility and attacked-set accuracy?
8 comments
Response to Reviewer 6uUH (2/2)
Part 2/2
Weakness 3: Accuracy across perturbation budgets
The author should discuss the robustness or performance of NOVA against different perturbation budgets.
Thank you for this suggestion. To examine this robustness, we designed budget sweeps for both conventional and generator attacks. We added budget sweeps on CIFAR-10 IPC10. EMN, LSP, and CWN are regenerated at 2, 4, 8, 16, and 32/255, and every point averages three synthesis seeds. The revised paper plots mean ± SD curves for CAFE, DM, IDM, and MTT in Fig. 4, and the appendix table also includes DM-HDD.
Table R3. Mean accuracy (%) over EMN, LSP, and CWN, reported as Vanilla / NOVA. Bold marks the better result.
| Backend | 2/255 | 8/255 | 32/255 |
|---|---|---|---|
| CAFE | 40.4 / 42.3 | 34.0 / 40.6 | 23.4 / 33.3 |
| DM | 41.4 / 42.2 | 37.8 / 39.6 | 27.8 / 30.0 |
| DM-HDD | 42.9 / 43.6 | 38.0 / 40.3 | 27.4 / 29.9 |
| IDM | 43.5 / 45.3 | 40.2 / 41.6 | 36.5 / 38.3 |
| MTT | 49.1 / 51.3 | 47.8 / 50.4 | 36.7 / 42.0 |
NOVA is higher at every budget for all five backends, and at 32/255 the gain ranges from 1.8 to 9.9 points. We also swept GUE from 8 to 64/255 and FUSE projected to 8, 16, and 32/255. On CIFAR-10 IPC10, NOVA matches or exceeds Vanilla in 47 of 48 cells. On four further dataset and IPC settings with all seeds completed, it wins 153, ties 17, and loses 22 of 192 cells, with a mean change of +4.5 points. The gain is smallest on CIFAR-100, so it is not uniform across every backend and budget.
The complete curves, the random-noise control, and the cross-dataset GUE/FUSE sweeps are in Appendix E.3.
Resonse to authors' rebuttal
I thank the authors for their detailed response and additional experiments. I appreciate their efforts in the rebuttal, which largely addresses my third concern.
However, my main concerns regarding the experimental evaluation remain insufficiently addressed.
1. Limited evaluation scale. Although the authors extend the evaluation to ImageNet-100 at 224×224 resolution, the experiment remains limited to a relatively small subset of ImageNet. Moreover, the absolute classification accuracy remains low, and the robustness improvement is relatively modest compared with the additional computational cost. Therefore, I am not fully convinced that the current results sufficiently demonstrate the practical significance and scalability of NOVA in larger-scale DD scenarios.
2. Insufficient comparison with existing defenses. My original concern was to compare NOVA with representative defense methods developed against unlearnable examples. However, the newly introduced baselines, including JPEG compression, grayscale conversion, and bit-depth reduction, are relatively simple image transformations rather than stronger, dedicated defenses proposed in the literature. While I acknowledge the distinction between producer-side and consumer-side defenses, this distinction does not eliminate the need for comparisons with more competitive existing methods. Consequently, the current experiments do not convince me of NOVA's advantages over established defenses.
Overall, while the additional experiments improve the paper, I believe the evaluation remains insufficient in terms of both scalability and competitive baselines. Therefore, my main concerns remain.
Response to Reviewer 6uUH (1/2)
Part 1/2
We thank the reviewer for the constructive comments. In response, we evaluated NOVA on 224x224 ImageNet-100, compared it with three consumer-side defenses against unlearnable examples across six DD backends, and added accuracy-versus-budget curves for several attacks and settings.
Q1: Larger, higher-resolution benchmark
We agree that our original benchmarks did not show whether NOVA remains effective at a realistic image resolution. We therefore distilled ImageNet-100 at 224x224 with DM, a scalable distribution-matching backend, at IPC10. Distillation runs for 10,000 iterations, and a ConvNetD5 consumer is trained for 1,000 epochs. All results average three independently distilled seeds.
Table R1. Accuracy (%) on ImageNet-100 (224x224), mean ± SD over three synthesis seeds. The last column averages EMN, LSP, and CWN. Bold marks the better result.
NOVA improves EMN, LSP, and CWN by 2.17, 1.30, and 3.00 points, and each gain holds in all three paired seeds. The attack mean rises by 2.15 points at a clean cost of 1.13 points. Distillation takes 11.3 GPU-hours for Vanilla and 18.1 for NOVA, a 1.59x overhead. The absolute IPC10 accuracy is still modest. We therefore read this result as evidence that the robustness gain carries over to 224x224 synthesis, not as a state-of-the-art ImageNet result.
We added this experiment and its cost to Appendix D.2, and the Limitations paragraph now states that our high-resolution evidence covers one backend.
Q2: Difference from unlearnable-example defenses
Thank you for this question. To make this difference concrete, we compared NOVA with three representative purification defenses across six backends. The two approaches act at different stages. UE defenses run on the consumer side and transform every received image after delivery, which can remove useful synthetic signal along with the perturbation. NOVA changes the producer-side distillation objective. It stabilizes each backend's native matching signal before delivery and leaves the delivered format and downstream training unchanged. The two can therefore be combined.
We compared NOVA with JPEG at quality 10, grayscale conversion, and 2-bit depth reduction (BDR2), each applied after the attack. The experiment covers CIFAR-10 IPC10, six backends, three synthesis seeds, and EMN, LSP, and CWN.
Table R2. Clean / three-attack mean accuracy (%). "None" is standard downstream training without purification. Bold marks the highest attacked accuracy per backend.
Averaged over the six backends, Vanilla without purification reaches 44.35 clean and 31.37 attacked accuracy. JPEG10, BDR2, and grayscale give 36.35/32.37, 40.33/32.23, and 33.77/31.52. The best purification therefore adds only 1.00 point under attack, while clean accuracy drops by 4.02 to 10.58 points. NOVA without purification reaches 44.37/36.52. It raises the attacked mean by 5.15 points at the same clean accuracy and is above every Vanilla+purification result on all six backends. Applying the same transformations to NOVA gives no consistent extra gain, which suggests that purification also removes useful distilled information.
The paper now explains this difference in intervention point, and Appendix G.1 reports all Vanilla and NOVA purification combinations.