acceptodds
Under review as a conference paper at ICLR 2027

Fragile by Distillation: Native-Objective Vicinal Alignment for Reliable Synthetic Sets

Abstract

Dataset distillation (DD) plays an important role in creating compact and efficient training sets for reducing downstream computational costs and enabling data sharing. However, previous works have overlooked a critical concern: the condensed datasets are vulnerable to malicious modifications during delivery. Our investigation shows that this weakness is prevalent across representative distillation methods and does not consistently diminish at larger distillation budgets. Models trained on these compromised sets suffer significant performance degradation. To bridge this gap, we introduce Native-Objective Vicinal Alignment (NOVA), a framework which can be incorporated into diverse matching-based DD approaches by the optimization of their native distillation objectives. The key innovation of NOVA lies in jointly matching the synthetic set and its perturbed neighbors to the same real-data target so that the matching holds across a neighborhood of the delivered set rather than at a single point. This matching reduces the shift in the native training signal induced by perturbation. Theoretical analysis and extensive experiments show that NOVA substantially improves the downstream performance of condensed datasets under malicious modifications while preserving clean utility and requiring no changes to the synthetic-set format or the downstream training pipeline. Across seven DD methods, five datasets, and three budgets, NOVA improves accuracy under different perturbations unseen during distillation in 96% of the evaluated settings, with limited impact on clean accuracy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dataset distillation, dataset condensation, availability attacks, adversarial perturbations, data poisoning, synthetic data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6uUHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses the vulnerabilities of dataset distillation to malicious perturbation and proposes NOVA to address this challenge. NOVA first jointly aligns clean and perturbed synthetic datasets to the same real-data target and ensures the matching holds across a neighborhood of the delivered set rather than at a single point. Additionally, this paper gives theoretical analysis along with experimental verification to demonstrate that NOVA improves the performance of models trained on a maliciously modified condensed dataset.

Critical strengths
  1. The paper discusses an important problem of defending against malicious modification of the distilled dataset.

  2. The proposed NOVA provides a straightforward framework to mitigate the perturbation by accommodating several matching objectives without changing the downstream targets.

Critical weaknesses
  1. The main concern is the scale of the evaluated dataset. Dataset distillation mainly works on large-scale datasets of high-resolution images to reduce training cost. The datasets discussed here are primarily based on small, low-resolution benchmarks. This raises my doubts about the practical significance of NOVA. It is recommended to evaluate NOVA on at least one or two scalable backends on a substantially larger benchmark to validate its practical significance.

  2. The objective of DD perturbation is quite similar to the definition of unlearnable examples. Lots of defense methods have been proposed to mitigate or purify the noise caused by unlearnable examples, thus enhancing the performance of the trained model. The author should clearly state the main difference of its defense and existing defense methods and, at a minimum, compare NOVA in the experiments with existing noising defense and mitigation methods.

  3. The author should discuss the robustness or performance of NOVA against different perturbation budgets. It is recommended to provide accuracy-versus-attack-budget curves for representative attacks and backends.

Questions for authors
  1. Can the authors evaluate NOVA using one or two scalable DD backends on a larger, higher-resolution benchmark to verify whether its robustness gains remain practically meaningful?

  2. What distinguishes NOVA from defenses against unlearnable examples, and how does it compare with representative mitigation or purification methods in terms of both clean-set utility and attacked-set accuracy?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 6uUH (1/2)

Part 1/2

We thank the reviewer for the constructive comments. In response, we evaluated NOVA on 224x224 ImageNet-100, compared it with three consumer-side defenses against unlearnable examples across six DD backends, and added accuracy-versus-budget curves for several attacks and settings.

Q1: Larger, higher-resolution benchmark

Can the authors evaluate NOVA using one or two scalable DD backends on a larger, higher-resolution benchmark to verify whether its robustness gains remain practically meaningful?

We agree that our original benchmarks did not show whether NOVA remains effective at a realistic image resolution. We therefore distilled ImageNet-100 at 224x224 with DM, a scalable distribution-matching backend, at IPC10. Distillation runs for 10,000 iterations, and a ConvNetD5 consumer is trained for 1,000 epochs. All results average three independently distilled seeds.

Table R1. Accuracy (%) on ImageNet-100 (224x224), mean ± SD over three synthesis seeds. The last column averages EMN, LSP, and CWN. Bold marks the better result.

ArmCleanEMNLSPCWNAttack mean
Vanilla DM14.87±0.276.18±0.731.62±0.0810.46±0.036.09
DM+NOVA13.73±0.448.35±1.472.92±0.0913.46±0.108.24

NOVA improves EMN, LSP, and CWN by 2.17, 1.30, and 3.00 points, and each gain holds in all three paired seeds. The attack mean rises by 2.15 points at a clean cost of 1.13 points. Distillation takes 11.3 GPU-hours for Vanilla and 18.1 for NOVA, a 1.59x overhead. The absolute IPC10 accuracy is still modest. We therefore read this result as evidence that the robustness gain carries over to 224x224 synthesis, not as a state-of-the-art ImageNet result.

We added this experiment and its cost to Appendix D.2, and the Limitations paragraph now states that our high-resolution evidence covers one backend.

Q2: Difference from unlearnable-example defenses

What distinguishes NOVA from defenses against unlearnable examples, and how does it compare with representative mitigation or purification methods in terms of both clean-set utility and attacked-set accuracy?

Thank you for this question. To make this difference concrete, we compared NOVA with three representative purification defenses across six backends. The two approaches act at different stages. UE defenses run on the consumer side and transform every received image after delivery, which can remove useful synthetic signal along with the perturbation. NOVA changes the producer-side distillation objective. It stabilizes each backend's native matching signal before delivery and leaves the delivered format and downstream training unchanged. The two can therefore be combined.

We compared NOVA with JPEG at quality 10, grayscale conversion, and 2-bit depth reduction (BDR2), each applied after the attack. The experiment covers CIFAR-10 IPC10, six backends, three synthesis seeds, and EMN, LSP, and CWN.

Table R2. Clean / three-attack mean accuracy (%). "None" is standard downstream training without purification. Bold marks the highest attacked accuracy per backend.

BackendVanilla, noneVanilla + JPEG10Vanilla + BDR2Vanilla + grayNOVA, none
CAFE42.7 / 22.434.8 / 28.636.9 / 26.434.2 / 31.842.3 / 36.7
DM41.8 / 28.635.3 / 30.938.0 / 29.932.7 / 30.142.4 / 31.4
DM-HDD43.3 / 28.236.0 / 30.438.5 / 29.133.6 / 30.343.7 / 31.6
IDC42.9 / 32.136.8 / 34.042.1 / 33.735.2 / 33.439.5 / 35.2
IDM46.4 / 36.337.0 / 36.144.7 / 38.034.2 / 32.446.9 / 38.5
MTT49.0 / 40.638.2 / 34.241.8 / 36.332.7 / 31.151.4 / 45.7

Averaged over the six backends, Vanilla without purification reaches 44.35 clean and 31.37 attacked accuracy. JPEG10, BDR2, and grayscale give 36.35/32.37, 40.33/32.23, and 33.77/31.52. The best purification therefore adds only 1.00 point under attack, while clean accuracy drops by 4.02 to 10.58 points. NOVA without purification reaches 44.37/36.52. It raises the attacked mean by 5.15 points at the same clean accuracy and is above every Vanilla+purification result on all six backends. Applying the same transformations to NOVA gives no consistent extra gain, which suggests that purification also removes useful distilled information.

The paper now explains this difference in intervention point, and Appendix G.1 reports all Vanilla and NOVA purification combinations.

AuthorsOpenReview ↗

Response to Reviewer 6uUH (2/2)

Part 2/2

Weakness 3: Accuracy across perturbation budgets

The author should discuss the robustness or performance of NOVA against different perturbation budgets.

Thank you for this suggestion. To examine this robustness, we designed budget sweeps for both conventional and generator attacks. We added budget sweeps on CIFAR-10 IPC10. EMN, LSP, and CWN are regenerated at 2, 4, 8, 16, and 32/255, and every point averages three synthesis seeds. The revised paper plots mean ± SD curves for CAFE, DM, IDM, and MTT in Fig. 4, and the appendix table also includes DM-HDD.

Table R3. Mean accuracy (%) over EMN, LSP, and CWN, reported as Vanilla / NOVA. Bold marks the better result.

Backend2/2558/25532/255
CAFE40.4 / 42.334.0 / 40.623.4 / 33.3
DM41.4 / 42.237.8 / 39.627.8 / 30.0
DM-HDD42.9 / 43.638.0 / 40.327.4 / 29.9
IDM43.5 / 45.340.2 / 41.636.5 / 38.3
MTT49.1 / 51.347.8 / 50.436.7 / 42.0

NOVA is higher at every budget for all five backends, and at 32/255 the gain ranges from 1.8 to 9.9 points. We also swept GUE from 8 to 64/255 and FUSE projected to 8, 16, and 32/255. On CIFAR-10 IPC10, NOVA matches or exceeds Vanilla in 47 of 48 cells. On four further dataset and IPC settings with all seeds completed, it wins 153, ties 17, and loses 22 of 192 cells, with a mean change of +4.5 points. The gain is smallest on CIFAR-100, so it is not uniform across every backend and budget.

The complete curves, the random-noise control, and the cross-dataset GUE/FUSE sweeps are in Appendix E.3.

Reviewer_6uUHOpenReview ↗

Resonse to authors' rebuttal

I thank the authors for their detailed response and additional experiments. I appreciate their efforts in the rebuttal, which largely addresses my third concern.

However, my main concerns regarding the experimental evaluation remain insufficiently addressed.

1. Limited evaluation scale. Although the authors extend the evaluation to ImageNet-100 at 224×224 resolution, the experiment remains limited to a relatively small subset of ImageNet. Moreover, the absolute classification accuracy remains low, and the robustness improvement is relatively modest compared with the additional computational cost. Therefore, I am not fully convinced that the current results sufficiently demonstrate the practical significance and scalability of NOVA in larger-scale DD scenarios.

2. Insufficient comparison with existing defenses. My original concern was to compare NOVA with representative defense methods developed against unlearnable examples. However, the newly introduced baselines, including JPEG compression, grayscale conversion, and bit-depth reduction, are relatively simple image transformations rather than stronger, dedicated defenses proposed in the literature. While I acknowledge the distinction between producer-side and consumer-side defenses, this distinction does not eliminate the need for comparisons with more competitive existing methods. Consequently, the current experiments do not convince me of NOVA's advantages over established defenses.

Overall, while the additional experiments improve the paper, I believe the evaluation remains insufficient in terms of both scalability and competitive baselines. Therefore, my main concerns remain.

Reviewer_WyUmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies malicious modification of a distilled dataset after its production and before downstream training. NOVA adds a high-loss neighbor to each distillation method’s native objective and trains the clean and perturbed synthetic sets against the same real-data target. Across seven methods and five datasets, the authors report improved downstream accuracy under five delivery-time modifications in most evaluated settings, with little average change in clean accuracy.

Critical strengths
  1. The paper identifies a meaningful vulnerability between dataset production and downstream use. The results show that modifying a compact synthetic set can substantially reduce its training utility across several distillation methods.
  2. NOVA uses each backend’s existing objective and leaves the delivered set’s format unchanged. The evaluation covers multiple matching families, datasets, and held-out attacks, with detailed per-IPC results that include unfavorable cases.
Critical weaknesses
  1. The introduction should define the research gap more precisely. Poisoned Distillation already studies modification of distilled sets during distribution, although its goal is a targeted backdoor rather than loss of general training utility. GUARD and ROME address test-time robustness, yet Table 2 shows that they also provide some protection here. The paper should explain at the mechanism level why these methods transfer to delivery-time attacks and what native-objective alignment contributes beyond them.
  2. The attack budgets are reported inconsistently. Section 5.1 states that all five attacks use 8/255, while Appendix D.1 reports 64/255 for GUE, 32/255 for LSP and CWN, and about 21.6/255 for FUSE. NOVA starts its inner search at zero and takes three steps of 2/255, reaching at most 6/255 under the stated settings. The gains against larger modifications may be valid, but their relation to the proposed local-neighborhood mechanism needs explanation. The different results for nominal 8/255 and 16/255 training radii in Table 6 also require clarification.
  3. Changes in the magnitude of the stored synthetic images remain an alternative explanation for part of the gain. The tensors are not clipped or quantized, and the paper reports no norm statistics, norm-matched Vanilla control, or evaluation after applying the same clipping and quantization to both sets. These checks are needed before attributing the gains specifically to stability of the native training signal.
  4. Comparable results are difficult to reconcile. The CAFE/CIFAR-10/IPC10 results differ across Tables 3–6 for apparently similar architectures and attacks. The supplementary CAFE launcher also defaults to NOVA weight 2 on EuroSAT, while Section 5.1 states that all main experiments use weight 1. The authors should identify the settings and runs behind the reported results. Several Vanilla accuracies are also well below previously reported values and need a protocol-level explanation.
  5. The evaluation does not include an attack designed to defeat NOVA. EMN is optimized on each delivered set, but the remaining attacks use generators or constructed patterns without adapting to the defense. The present results establish an advantage against those modifications, not against an adversary who knows NOVA’s procedure and targets downstream training accuracy.
  6. Attacked-accuracy gains are sometimes difficult to separate from clean-accuracy gains. For MTT on CIFAR-10 and SVHN, NOVA improves clean accuracy more than attacked accuracy, so the clean-to-attacked gap widens. Both absolute attacked accuracy and the change in degradation relative to each set’s clean accuracy should be reported when describing robustness.
  7. The breadth of the results should not be read as independent statistical evidence. Multiple attack cells share the same distilled set, main tables lack uncertainty estimates, and IPC averages conceal individual regressions. The comparison with equally small real subsets is also limited to a few methods and one attack under a separate downstream training recipe.

Minor:

  1. The theoretical analysis motivates NOVA but depends on an unmeasured relationship between native-signal deviation and downstream training. It does not establish a quantitative guarantee for the reported accuracy gains.
  2. The text overstates some trends. The claim that every method suffers a large drop on every dataset has small-drop exceptions, and the stated 0.7-point bound on clean-accuracy changes across depths does not hold for MTT at depth 2.
  3. The supplementary package implements NOVA for CAFE but does not provide the other six backend integrations. This limits inspection of the claimed backend-independent implementation.
Questions for authors
  1. How does NOVA’s availability objective differ from prior distribution-stage backdoor work? What evidence explains why GUARD and ROME help in this setting and isolates the additional contribution of native-objective alignment?
  2. How should the common 8/255 budget in Section 5.1 be reconciled with the realized attack budgets in Appendix D.1? What inner-search settings produced the 16/255 result in Table 6?
  3. Which runs produced the differing CAFE results across Tables 3–6? Did the main EuroSAT experiments use NOVA weight 1, as stated in the paper, or weight 2, as set by the supplementary launcher?
  4. Do the gains persist with a norm-matched Vanilla set and after identical clipping or quantization of both sets?
  5. Does NOVA remain effective against an attacker who knows the method and optimizes the delivered images to reduce downstream accuracy? For MTT, how much of the gain remains when measured relative to each set’s clean accuracy?
4 comments
AuthorsOpenReview ↗

Response to Reviewer WyUm (1/4)

Part 1/4

We sincerely thank the reviewer for the careful and constructive review. In response, we added common-consumer baselines, a controlled training-radius sweep, budget-controlled and adaptive attacks, amplitude and format controls, and a degradation-reduction metric. We also corrected the attack budgets, the provenance of each table, and several overstated claims. Below we address each point, and we are happy to clarify further.

Q1 & Weakness 1

How does NOVA’s availability objective differ from prior distribution-stage backdoor work? What evidence explains why GUARD and ROME help in this setting and isolates the additional contribution of native-objective alignment?

Thank you for this helpful suggestion. We are sorry that we did not explain this research gap clearly in the paper. To clarify this gap, we designed a matched-parent comparison, a common-consumer comparison, and a compute-matched anchoring control. Below, we show how NOVA is different from earlier distribution-stage work and robust-DD baselines. We have addressed this previously unclarified research gap in the Introduction. of our method.

  • (i) Matched-parent transfer. Relative to its own parent pipeline, NOVA gives the largest gain on both datasets. ROME has the highest absolute accuracy on CIFAR-10.

    Table R1. Accuracy (%) at IPC10 over five attacks. Gain over parent is the five-attack mean of each defense minus that of its own parent pipeline without the defense. Bold marks the best result per dataset.

    DatasetDefenseCleanAttack meanWorst attackGain over parent
    CIFAR-10NOVA (CAFE)42.4137.2833.20+14.61
    CIFAR-10GUARD24.6921.8818.11+0.30
    CIFAR-10ROME44.3541.8439.04+2.95
    CIFAR-100NOVA (CAFE)23.0822.8821.49+4.37
    CIFAR-100GUARD24.5719.8614.77+1.17
    CIFAR-100ROME18.9118.8918.67+1.01

    GUARD and ROME aim to make models trained on the distilled set robust to test-time input perturbations. Their synthetic images therefore already carry some smoothness, which gives partial protection when the stored images are modified. NOVA instead stabilizes the native matching signal that produces the synthetic set, which gives the largest gain over its own parent.

  • (ii) Common-consumer comparison. We trained one common ConvNet consumer on CIFAR-100 IPC10. RDC has the highest clean accuracy and wins three attacks, but CAFE+NOVA has the highest mean and worst-case accuracy.

    Table R2. Accuracy (%) under one common consumer on CIFAR-100 IPC10. Bold marks the best result.

    MethodCleanEMNGUEFUSELSPCWNMeanWorst
    ROME18.218.518.618.418.218.318.418.2
    RDC parent30.228.219.728.527.517.424.317.4
    RDC30.828.921.029.628.017.625.017.6
    CAFE26.821.216.826.218.724.521.516.8
    CAFE+NOVA26.025.423.525.725.325.825.123.5

    ROME is nearly unaffected by attacks but has much lower accuracy. We therefore do not claim that NOVA dominates these methods on every metric. GUARD is omitted here because no compatible released artifact was available.

  • (iii) What target anchoring adds. Native Ascent Consistency uses the same ascent steps and native-loss evaluations as NOVA, but it only aligns the perturbed synthetic signal with the clean one. NOVA instead anchors both signals to the real-data target, which improves clean accuracy and the attack mean in all three settings.

    Table R3. Compute-matched comparison on CIFAR-10, reported as Native Ascent Consistency / NOVA (%). The attack mean averages EMN, LSP, and CWN. Bold marks the better result.

    SettingCleanEMNLSPCWNAttack mean
    CAFE IPC1041.7 / 42.037.4 / 36.530.3 / 31.639.8 / 40.335.8 / 36.1
    CAFE IPC5050.8 / 52.048.5 / 50.232.7 / 36.045.6 / 48.042.3 / 44.7
    DM IPC1041.9 / 42.537.1 / 37.821.5 / 22.234.1 / 35.230.9 / 31.7

    Consistency is already strong because a clean signal near the target acts as an indirect anchor. The explicit anchor further prevents the clean and perturbed signals from drifting together.

We have narrowed the novelty claim and discussed Poisoned Distillation in the Introduction, revised Sec. 5.2 and Tables 2–3, and added the per-attack results and anchoring controls in the revised manuscript (Appendix E.5 and F.1).

AuthorsOpenReview ↗

Response to Reviewer WyUm (2/4)

Part 2/4

Q2 & Weakness 2

How should the common 8/255 budget in Section 5.1 be reconciled with the realized attack budgets in Appendix D.1? What inner-search settings produced the 16/255 result in Table 6?

We acknowledge that Sec. 5.1 did not clearly separate NOVA's training radius from the attack budgets. To explain this relation, we designed a controlled radius sweep, budget-controlled attacks, and a random-noise control. Only NOVA's training radius is 8/255. The evaluation attacks use the default budgets of their original papers, which are 8/255 for EMN, about 21.6/255 for FUSE after its 2.7× output scaling, 32/255 for LSP and CWN, and 64/255 for GUE. The inner search starts at zero and takes three steps of 2/255, so it reaches at most 6/255. Robustness at 16–64/255 is therefore empirical transfer rather than a guarantee. The submitted 16/255 setting also used a step of 4/255, so radius and step size changed together.

  • (i) Controlled radius sweep. We re-distilled CIFAR-100 IPC10 with three steps of size ε/4 at ε = 4, 8, and 16/255. The best radius depends on the backend. CAFE peaks near 8/255, DM benefits from 16/255, and IDC trades clean accuracy for robustness to the adaptive attack. We therefore keep 8/255 as a common default.

    Table R4. Results on CIFAR-100 IPC10, reported as clean accuracy / mean over EMN, LSP, GUE, and CWN / unrolled attack at 8/255 from Part 4. Bold marks the best attacked results.

    BackendVanillaNOVA ε=4NOVA ε=8NOVA ε=16
    CAFE24.4 / 17.2 / 12.125.3 / 17.8 / 12.423.4 / 22.8 / 22.124.8 / 22.9 / 20.5
    DM23.2 / 10.8 / 4.522.9 / 13.3 / 7.423.0 / 15.0 / 8.222.4 / 16.6 / 9.6
    IDC24.7 / 13.2 / 5.122.1 / 15.8 / 8.820.6 / 15.9 / 12.019.1 / 15.5 / 14.3
  • (ii) Budget-controlled attacks. We evaluated GUE at 8–64/255 and FUSE projected to 8, 16, and 32/255 on CIFAR-10 IPC10 for six backends. NOVA matches or outperforms Vanilla in 47 of the 48 backend, attack, and budget combinations.

    Table R5. Accuracy (%) under FUSE projected to a fixed ℓ∞\ell_\infty budget, reported as Vanilla / NOVA. Bold marks the better result.

    Backend8/25516/25532/255
    CAFE37.6 / 42.027.8 / 40.719.3 / 34.7
    DM37.4 / 39.230.7 / 33.022.1 / 23.9
    DM-HDD37.6 / 40.129.9 / 33.220.8 / 23.2
    IDM44.5 / 45.941.1 / 43.233.9 / 37.1
    IDC34.9 / 38.629.1 / 35.524.2 / 30.2
    MTT47.9 / 50.844.0 / 48.232.3 / 39.5
  • (iii) Noise control and native-loss link. At 32/255, random-sign noise lowers Vanilla accuracy by only 1.7–3.9 points, whereas structured attacks lower it by up to 27.9 points. Across four sweeps, the relative increase in native matching loss predicts the downstream accuracy drop with Spearman ρ\rho = 0.95–0.99. This is empirical support, not a quantitative guarantee.

We have corrected all budgets in Sec. 5.1, replaced the radius interpretation of the sensitivity results in Table 23 with the controlled sweep, narrowed Sec. 3 and the Limitations, and added the budget and noise curves and the radius sweep in the revised manuscript (Appendix E.3 and F.2).

Weakness 7 & Minor

The breadth of the results should not be read as independent statistical evidence.

The text overstates some trends.

We appreciate this careful check. To check these points, we re-examined the statistical scope, compared distilled sets with equally small real subsets, and revised the overstated statements.

  • (i) Statistical scope. The main Vanilla/NOVA grid averages three synthesis seeds with three downstream trainings each, and the Appendix reports the per-IPC results. The recipe, compute-matched, and λ\lambda controls report mean ± SD (Appendix D.2, F.1, and F.2), and the budget curves show ± SD. Because attacks on one set are correlated, we no longer count attack cells as independent samples.
  • (ii) Real-subset comparison. We compare 100-image CAFE, DM, and MTT sets on CIFAR-10 with class-balanced random and teacher-feature representative real subsets of the same size, using the same ConvNet initialization and training recipe. At the realized FUSE budget of 22/255, the distilled sets lose more accuracy than the real subsets, whereas random-sign noise has a much smaller effect. We read this as evidence of sensitivity beyond sample count in this setting, not as proof that distillation is the only cause.

We have revised "every method suffers a large drop on every dataset" to "the large majority of evaluated pairs", corrected the 0.7-point depth bound, which does not hold for MTT at depths 2 and 3, and added a matched small-subset control (Appendix E.7). We will release the code for all seven backends after the review.

AuthorsOpenReview ↗

Response to Reviewer WyUm (3/4)

Part 3/4

Q3 & Weakness 4

Which runs produced the differing CAFE results across Tables 3–6? Did the main EuroSAT experiments use NOVA weight 1, as stated in the paper, or weight 2, as set by the supplementary launcher?

Several Vanilla accuracies are also well below previously reported values and need a protocol-level explanation.

Thank you for this careful check. To clarify the provenance, we traced each table to its experiment group and re-ran CAFE on EuroSAT with both λ\lambda values. All backends are evaluated with the same ConvNetD3 consumer, so the differences do not come from the evaluation network. The main grid, the sensitivity study, and the compute-matched ablation are three independently distilled groups with three synthesis seeds each, and every Vanilla/NOVA pair is matched within its own group.

  • (i) NOVA weight on EuroSAT. The CAFE EuroSAT row of the main grid used λ=2\lambda=2 instead of the stated λ=1\lambda=1, so we re-ran both values under a matched pipeline.

    Table R7. CAFE accuracy (%) on EuroSAT IPC10. Bold marks the best result.

    VariantCleanEMNLSPCWN
    Vanilla54.734.521.630.8
    NOVA, λ=1\lambda=154.542.329.247.7
    NOVA, λ=2\lambda=254.145.431.949.7

    With λ=1\lambda=1, NOVA already improves EMN, LSP, and CWN by 7.8, 7.6, and 16.9 points, which is about 80% of the gain obtained with λ=2\lambda=2. The extra gain at λ=2\lambda=2 matches CAFE's amplitude sensitivity shown below, as the RMS rises from 1.04 to 1.27. The conclusion does not depend on λ\lambda.

  • (ii) Vanilla baselines. The lower baselines mainly reflect the shortened synthesis schedules of the seven-backend grid, and NOVA's gains persist under official recipes. Training on the same sets for 1000 instead of 100 epochs changes clean accuracy by only 0.8–4.1 points. With a stale MTT IPC50 entry replaced, Vanilla MTT reaches 49.0, 52.5, and 61.4 at IPC10, 30, and 50.

    Table R8. Accuracy (%) under stronger recipes on CIFAR-10 IPC10 as Vanilla / NOVA, with the attack mean over EMN, LSP, and CWN. Bold marks the better one.

    BackendReportedCleanAttack mean
    CAFE (official)46.345.36 / 45.7221.40 / 38.95
    IDM (official)58.657.05 / 55.8647.74 / 54.00
    IDC (2000 iterations)67.555.21 / 51.2241.66 / 45.87

We have corrected λ\lambda, stated the experiment group in the captions and λ\lambda in Sec. 5.1 and Appendix D.2, clarified the protocol in Sec. 5.1, and added these results (Appendix D.2 and F.2).

Q4 & Weakness 3

Do the gains persist with a norm-matched Vanilla set and after identical clipping or quantization of both sets?

Thank you for this insightful suggestion. To test this explanation, we designed amplitude and format controls. We rescale each set around the dataset mean as p′=μ+r(p−μ)p'=\mu+r(p-\mu). V↑V^{\uparrow} matches Vanilla to NOVA's centered RMS, N↓N^{\downarrow} matches NOVA to Vanilla's, and attacks are regenerated afterwards.

  • (i) Bidirectional amplitude matching. Over 6 backends and 6 conditions, 72% of NOVA's gain under attack remains after scaling Vanilla up, and 68% remains after scaling NOVA down.

    Table R9. Attacked accuracy (%) at IPC10, averaged over four datasets and three attacks. Bold marks the better result.

    BackendRMS N/VN/VVV / N↓N^{\downarrow}V↑V^{\uparrow} / NN
    CAFE2.0–2.8×21.5 / 24.330.0 / 33.8
    DM1.0–1.3×22.5 / 28.323.5 / 30.7
    DM-HDD1.0–1.2×23.0 / 28.224.0 / 30.1
    IDC1.1–1.4×28.1 / 33.930.7 / 36.1
    IDM1.0–1.1×34.0 / 37.734.6 / 38.5
    MTT1.0–1.2×31.3 / 33.833.0 / 35.4
    Mean–26.7 / 31.129.3 / 34.1
  • (ii) RMS-capped synthesis. For CAFE, we re-distill while projecting the RMS to the Vanilla value after every update. At identical RMS, NOVA still improves EMN, LSP, and CWN by 8.1, 9.7, and 9.5 points at IPC10 and by 16.4, 7.9, and 8.9 points at IPC50, at clean costs of 1.6 and 8.4 points.

  • (iii) Clipping and 8-bit quantization. Both transforms give nearly identical results, and NOVA keeps its advantage on all six backends on CIFAR-10 IPC10.

    Table R10. Accuracy (%) after clipping on CIFAR-10 IPC10, Vanilla / NOVA.

    BackendCAFEDMDM-HDDIDMIDCMTT
    Clean40.5 / 35.041.6 / 42.443.2 / 43.546.3 / 46.943.6 / 41.545.7 / 46.0
    Attack mean22.1 / 27.428.5 / 30.928.2 / 31.436.3 / 38.531.8 / 35.135.9 / 37.8

We have stated that amplitude explains part but not all of the gains, added these results (Appendix E.2, F.1, and G.1), and reported the clipping cost in the Limitations.

AuthorsOpenReview ↗

Response to Reviewer WyUm (4/4)

Part 4/4

Q5 & Weaknesses 5–6

Does NOVA remain effective against an attacker who knows the method and optimizes the delivered images to reduce downstream accuracy? For MTT, how much of the gain remains when measured relative to each set’s clean accuracy?

We thank the reviewer for these in-depth questions, which led us to further strengthen the evaluation of NOVA. We add two adaptive attacks that are optimized against each delivered set separately, and a metric that removes the clean-accuracy gain.

  • (i) Native-objective PGD. The attacker maximizes the backend's own matching loss on each set with 200 PGD steps and 5 random restarts. NOVA outperforms Vanilla in all 16 settings, and the advantage holds in 47 of 48 individual seed pairs. The margins become smaller for the DM family at 16/255.

    Table R11. Accuracy (%) under native-objective PGD at IPC10, reported as Vanilla / NOVA. Bold marks the better result.

    DatasetBackend8/25516/255
    CIFAR-100CAFE17.0 / 22.99.3 / 22.1
    CIFAR-100DM7.2 / 12.42.9 / 5.4
    CIFAR-100DM-HDD8.1 / 14.63.4 / 5.2
    CIFAR-100IDM10.4 / 13.34.5 / 5.2
    SVHNCAFE12.0 / 18.611.7 / 12.2
    SVHNDM11.9 / 24.310.4 / 12.1
    EuroSATCAFE20.2 / 37.617.4 / 26.6
    EuroSATDM19.4 / 25.514.7 / 19.8
  • (ii) Attack through downstream training. The attacker unrolls the SGD steps of the victim on the perturbed set and uses four surrogate models. At 8/255, NOVA is better in 16 of 18 seed pairs on CIFAR-100 and in 32 of 36 on SVHN and EuroSAT. IDM and MTT show no clear gain on CIFAR-100, and at 16/255 or above most backends approach chance accuracy in both arms. A stronger attack with 1000 steps, 16 unrolled steps, and 8 surrogates gives the same CAFE result of 11.6 for Vanilla and 21.7 for NOVA.

    Table R12. Accuracy (%) under the unrolled attack on CIFAR-100 IPC10, reported as Vanilla / NOVA. Bold marks the better result.

    Backend8/25516/25532/255
    CAFE12.1 / 22.15.4 / 16.22.1 / 7.7
    DM4.5 / 8.22.7 / 2.41.2 / 1.4
    DM-HDD4.2 / 7.02.0 / 2.51.4 / 1.6
    IDM8.4 / 9.04.8 / 4.62.3 / 2.9
    IDC5.1 / 12.02.4 / 3.51.0 / 0.9
    MTT26.6 / 26.520.4 / 21.610.6 / 12.6
  • (iii) Degradation reduction. We report R=(Natt−Vatt)−(Nclean−Vclean)R=(N_{\rm att}-V_{\rm att})-(N_{\rm clean}-V_{\rm clean}), the gain under attack minus the clean gain. R>0R>0 means that NOVA reduces the drop from clean to attacked accuracy. CAFE and IDC have R>0R>0 in every setting. The DM family and IDM are close to zero on CIFAR-10 IPC30 and IPC50. For MTT, the reviewer is right that much of the gain comes from higher clean accuracy.

    Table R13. RR (points) per backend over 6 settings and 3 attacks (EMN, LSP, and CWN).

    CAFEDMDM-HDDIDMIDCMTT
    Cells with R>0R>018/1815/1817/1817/1818/1813/18
    Range+1.5 to +29.9−0.5 to +32.10.0 to +23.7−0.6 to +15.5+3.7 to +36.6−7.0 to +9.1
    Mean+15.7+6.3+5.6+3.7+11.6+1.1

    Table R14. MTT gains (points) averaged over EMN, LSP, and CWN. Bold marks R>0R>0.

    SettingGain under attackClean gainRR
    CIFAR-10 IPC10+5.1+2.4+2.7
    CIFAR-10 IPC30+8.2+8.8−0.6
    CIFAR-10 IPC50+9.3+2.2+7.1
    SVHN IPC10+6.5+6.2+0.3
    EuroSAT IPC10+4.0+7.5−3.5

These are finite empirical attacks rather than certificates. We have added both adaptive attacks to Sec. 5.3, reported RR alongside absolute accuracy in Appendix E.4, and removed the claim that every gain under attack reduces the robustness gap. The full native-PGD, unrolled-attack, strength, and per-seed results are added in the revised manuscript (Appendix E.4).

Reviewer_Lic3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that distilled datasets are fragile to perturbations added after distillation, e.g. when the synthetic set is shared and modified before someone trains on it. The authors attribute this to DD objectives only constraining the matching loss at the final set, not around it. They propose NOVA, which is basically adversarial training on each method's own matching loss: find a worst-case perturbation of the synthetic set with a few PGD steps, and require both the clean and perturbed sets to match the real-data target. It is applied to seven DD methods on five datasets and improves accuracy under several held-out attacks in nearly all settings, with little change in clean accuracy.

Critical strengths

The experiments are broad: seven DD methods covering all the main matching families, five datasets, several IPCs, and attacks that are not seen during distillation. The improvement is quite consistent across all of these. The method is also simple and plugs into existing pipelines without changing the output format. The problem itself (robustness of the distilled set as an artifact) is a reasonable thing to point out to the DD community.

Critical weaknesses
  1. I'm not sure the gains come from the proposed mechanism. The main text says all attacks use 8/255, but according to the appendix most of them are actually much larger (32/255 or more). So NOVA, trained at 8/255, is helping against much bigger perturbations, which the local analysis in Sec. 3 doesn't really explain. One simple explanation I'd want ruled out is that NOVA just increases the magnitude/contrast of the synthetic images (Figs. 9-11 suggest this), which makes any fixed-size perturbation relatively weaker. A Vanilla set rescaled to the same amplitude would be an easy baseline. There is also no adaptive attack, e.g. running the same PGD on the matching loss with more steps against the NOVA set. The threat model also needs more motivation: if someone can modify the file, a checksum would catch it, and the small-budget constraint is hard to justify for images that don't look natural anyway.

  2. Some of the baseline numbers look low compared to the original papers. IDC and MTT on CIFAR-10 in particular are far below what was originally reported, and MTT even gets worse at higher IPC. This makes me suspect a shortened evaluation recipe, and I'm not sure the gains would hold on properly trained sets. The same CAFE setting also gives noticeably different numbers in different tables (e.g. Table 3 vs Table 5). In Table 2, RDC's base model is near chance, and ROME actually has higher absolute attacked accuracy than NOVA, which the "gain over own base" framing hides.

Questions for authors
  1. What happens if you rescale the Vanilla set to the same pixel magnitude as the NOVA set? If most of the gain remains, I'd be happy to raise my score.
  2. What evaluation protocol is used (epochs, augmentation)? Why are some Vanilla results well below the published numbers?
  3. How does NOVA do against an adaptive attack, e.g. stronger PGD on the matching loss itself?
  4. Why do the tables report different results for the same CAFE / CIFAR-10 / IPC10 setting?
  5. (Clarification) In the ablation, does Native Consistency use the same number of ascent steps as NOVA? If not, the comparison doesn't isolate the effect of anchoring to the target.
8 comments
AuthorsOpenReview ↗

Response to Reviewer Lic3 (1/4)

Part 1/4

We greatly appreciate your constructive and detailed feedback! Below are our responses to your questions and concerns. We are happy to provide additional clarification if needed.

Q1: rescale the Vanilla set to the same pixel magnitude as the NOVA set

What happens if you rescale the Vanilla set to the same pixel magnitude as the NOVA set?

Thank you for this suggestion. To test whether magnitude explains the gain, we designed two amplitude controls, bidirectional RMS matching and RMS-capped synthesis. We conducted bidirectional amplitude matching to test whether NOVA's gains can be explained by pixel magnitude. Let VV and NN denote the original Vanilla and NOVA sets. We rescale an image around the fixed dataset mean μ\mu as p′=μ+r(p−μ).p' = \mu + r(p-\mu). V↑V^{\uparrow} denotes Vanilla rescaled to NOVA's centered pixel RMS, and N↓N^{\downarrow} denotes NOVA rescaled to Vanilla's. Matching factors are computed only from the clean delivered artifacts. We regenerate all attacks after rescaling, including re-optimizing EMN, and use the same downstream protocol for all four arms.

Table R1 covers all six backends on the four datasets of our main grid at IPC10: CIFAR-10, CIFAR-100, SVHN, and EuroSAT. Each column pair reports one dataset, and the CIFAR-10 IPC30 and IPC50 conditions are added in the retained fractions below.

Table R1. Three-attack average (%) of EMN/LSP/CWN on CIFAR-10, CIFAR-100, SVHN, and EuroSAT at IPC10 under matched amplitude. Each cell compares Vanilla and NOVA at the same RMS, and Mean averages the four datasets: VV / N↓N^{\downarrow} at Vanilla's RMS and V↑V^{\uparrow} / NN at NOVA's RMS. Bold marks the better result.

BackendCIFAR-10 VV / N↓N^{\downarrow}CIFAR-10 V↑V^{\uparrow} / NNCIFAR-100 VV / N↓N^{\downarrow}CIFAR-100 V↑V^{\uparrow} / NNSVHN VV / N↓N^{\downarrow}SVHN V↑V^{\uparrow} / NNEuroSAT VV / N↓N^{\downarrow}EuroSAT V↑V^{\uparrow} / NNMean VV / N↓N^{\downarrow}Mean V↑V^{\uparrow} / NN
CAFE22.4 / 26.131.8 / 37.118.4 / 20.823.4 / 23.216.3 / 18.827.1 / 32.529.1 / 31.737.8 / 42.521.5 / 24.330.0 / 33.8
DM29.3 / 30.629.3 / 31.511.8 / 15.113.0 / 16.421.8 / 37.223.2 / 41.827.1 / 30.228.3 / 33.022.5 / 28.323.5 / 30.7
DM-HDD28.3 / 30.729.0 / 31.912.6 / 16.313.6 / 17.123.2 / 34.924.7 / 38.927.9 / 31.028.7 / 32.523.0 / 28.224.0 / 30.1
IDM36.1 / 38.336.4 / 38.415.8 / 16.715.9 / 16.949.4 / 58.350.4 / 59.834.6 / 37.435.7 / 38.934.0 / 37.734.6 / 38.5
IDC32.0 / 34.632.7 / 35.213.1 / 14.614.6 / 15.736.6 / 52.442.1 / 57.030.8 / 34.133.2 / 36.628.1 / 33.930.7 / 36.1
MTT40.6 / 43.843.3 / 45.728.2 / 27.728.5 / 27.924.9 / 29.427.2 / 32.631.5 / 34.532.7 / 35.431.3 / 33.833.0 / 35.4

At equal amplitude, NOVA has the higher attacked average in 45/48 paired comparisons. Per attack, it stays ahead in 67 and 65 of the 72 paired settings at the Vanilla and NOVA RMS (Table 4). We also re-distilled CAFE, which has the largest RMS change, while projecting its RMS to the Vanilla value after every update. At exactly matched RMS, this capped NOVA still improves EMN/LSP/CWN by +8.1/+9.7/+9.5 points at IPC10, at a clean cost of 1.6 points (Appendix F.1). Summed over 6 backends × 6 conditions (CIFAR-10 IPC10/30/50, CIFAR-100 IPC10, SVHN IPC10, EuroSAT IPC10) and averaged equally across backends, 72% of NOVA's attacked-accuracy gain remains after scaling Vanilla up and 68% after scaling NOVA down. Averaged over the four IPC10 datasets, NOVA improves attacked accuracy by 7.4 points without rescaling, 4.3 points at the Vanilla RMS, and 4.8 points at the NOVA RMS. Amplitude thus contributes to the results but does not explain the overall improvement. The summary is in Table 4 of Sec. 5.3, and Table R1, the per-attack results, and the retained fractions are in Appendix E.2.

AuthorsOpenReview ↗

Response to Reviewer Lic3 (2/4)

Part 2/4

Q2: Evaluation protocol, low baselines

What evaluation protocol is used (epochs, augmentation)? Why are some Vanilla results well below the published numbers?

We apologize for not stating the protocol clearly. To clarify the protocol, we designed three checks on evaluation length, synthesis recipe, and the MTT evaluation. The main grid uses shortened synthesis schedules for the seven-backend, five-dataset comparison and evaluates all sets with ConvNet (100 epochs, lr 0.01, each backend's native augmentation and decoding; DATM uses its native 1000-epoch DSA+ZCA protocol), with identical settings for Vanilla and NOVA. We therefore verified that NOVA's gains hold when each method follows its original settings and reaches its reported accuracy (CIFAR-10 IPC10 unless noted).

  • (i) Evaluation length. Training the same frozen sets for 1000 rather than 100 epochs raises Vanilla clean accuracy by only 0.8–4.1 points.

    Table R2. Vanilla clean accuracy (%) under 100 vs. 1000 evaluation epochs.

    EpochsCAFEDMDM-HDDIDMIDCMTT
    10042.441.943.046.342.649.0
    100043.942.745.048.443.653.1
  • (ii) Stronger synthesis recipes. We reran CAFE and IDM with their official recipes and IDC with 2000 synthesis iterations and its 1000-epoch recipe. Vanilla approaches published accuracy, and NOVA still improves the attacked average by 17.55 (CAFE), 6.27 (IDM), and 4.21 (IDC) points, with clean costs of 1.19 (IDM) and 3.99 (IDC).

    Table R3. Accuracy (%) under stronger recipes (Vanilla / NOVA); Att. averages EMN/LSP/CWN. Bold marks the better Att.

    BackendPublishedCleanEMNLSPCWNAtt.
    CAFE (official)46.345.36 / 45.7224.22 / 39.9114.20 / 32.9125.79 / 44.0421.40 / 38.95
    IDM (official)58.657.05 / 55.8635.99 / 50.7355.24 / 55.5951.98 / 55.6947.74 / 54.00
    IDC (2000 iters)67.555.21 / 51.2230.87 / 37.1151.53 / 50.4042.58 / 50.0941.66 / 45.87
  • (iii) Corrected MTT. The previous IPC50 entry came from a stale evaluation. Re-evaluating the frozen MTT checkpoints gives Vanilla clean accuracy of 49.0±1.8, 52.5±13.0, and 61.4±6.0 at IPC10/30/50, which increases with IPC.

We have clarified the protocol in Sec. 5.1, corrected MTT, and added these results (Appendix D.2).

Weakness 1: Attack budgets, theoretical scope, and threat model

The main text says all attacks use 8/255, but according to the appendix most of them are actually much larger (32/255 or more).

We apologize for this incorrect statement in Sec. 5.1. Only NOVA's training radius is 8/255. The evaluation attacks use their original default budgets (EMN 8, FUSE ≈22 after its 2.7× output scaling, LSP/CWN 32, GUE 64, all /255). To separate budget effects from NOVA's gains, we evaluated budget-controlled attacks and a random-noise control on CIFAR-10 IPC10.

  • (i) Budget-controlled attacks. We evaluated GUE at 8–64/255 and FUSE projected to 8/16/32/255. NOVA matches or outperforms Vanilla in 47 of 48 budget cells, and at NOVA's 8/255 training radius, projected FUSE gains are +1.4 to +4.4 points.

    Table R4. Accuracy (%) under FUSE projected to a fixed ℓ∞\ell_\infty budget. Bold marks the better result.

    Backend8/255 Vanilla8/255 NOVA16/255 Vanilla16/255 NOVA32/255 Vanilla32/255 NOVA
    CAFE37.642.027.840.719.334.7
    DM37.439.230.733.022.123.9
    DM-HDD37.640.129.933.220.823.2
    IDM44.545.941.143.233.937.1
    IDC34.938.629.135.524.230.2
    MTT47.950.844.048.232.339.5
  • (ii) Random-noise control. At 32/255, random noise lowers Vanilla accuracy by only 1.7–3.9 points, versus up to 27.9 for structured attacks, so the degradation is driven by attack structure, not perturbation magnitude alone.

    Table R5. Vanilla accuracy (%) at 32/255.

    BackendCleanRandom noiseLSPCWN
    CAFE42.740.514.827.5
    DM41.838.819.431.2
    DM-HDD43.339.419.731.1
    IDM46.443.942.636.2
    MTT49.047.330.644.7
  • (iii) Theoretical scope. Sec. 3 motivates optimization within the local 8/255 neighborhood and does not guarantee robustness at 16–64/255. Empirically, the relative increase in native matching loss strongly predicts the accuracy drop (Spearman ρ\rho = 0.95–0.99 on CAFE IPC10/30/50 and DM IPC10).

We have corrected the budgets in Sec. 5.1, narrowed the theoretical claim in Sec. 3 and the Limitations, and added the dose curves and random-noise control (Appendix E.3).

AuthorsOpenReview ↗

Response to Reviewer Lic3 (3/4)

Part 3/4

Q3: Adaptive attack on the native matching loss

How does NOVA do against an adaptive attack, e.g. stronger PGD on the matching loss itself?

Thank you for this suggestion. To answer this question, we designed an adaptive attack that knows NOVA and targets each set's own matching loss. We attacked each delivered set by directly maximizing its own native matching loss, using 200-step PGD with five random restarts. Vanilla and NOVA are therefore each attacked by perturbations optimized against their own artifacts.

  • (i) Adaptive native-loss PGD. NOVA outperforms Vanilla in all 16 settings, and the advantage holds in 47 of 48 individual seed pairs.

    Table R6. Accuracy (%) under adaptive native-loss PGD at IPC10, reported as Vanilla / NOVA. Bold marks the better result.

    DatasetBackend8/25516/255
    CIFAR-100CAFE17.0 / 22.99.3 / 22.1
    CIFAR-100DM7.2 / 12.42.9 / 5.4
    CIFAR-100DM-HDD8.1 / 14.63.4 / 5.2
    CIFAR-100IDM10.4 / 13.34.5 / 5.2
    SVHNCAFE12.0 / 18.611.7 / 12.2
    SVHNDM11.9 / 24.310.4 / 12.1
    EuroSATCAFE20.2 / 37.617.4 / 26.6
    EuroSATDM19.4 / 25.514.7 / 19.8
  • (ii) Threat model. A trusted signature or authenticated checksum should be preferred whenever a verifiable original is available, and NOVA is not intended to replace it. NOVA targets mirrored, re-hosted, converted, or otherwise redistributed synthetic sets for which the consumer has no authenticated reference. The budget constraint follows the unlearnable-example literature and keeps the modification small relative to the stored content. Because synthetic images need not look natural, we do not rely on a single budget. The sweeps from 2/255 to 64/255 in Weakness 1 and Appendix E.3 show that NOVA keeps its advantage across budgets.

We have added the adaptive-attack results to Sec. 5.3 and Appendix E.4, and clarified the threat model in the Limitations.

Q4: Different CAFE results across tables

Why do the tables report different results for the same CAFE / CIFAR-10 / IPC10 setting?

We apologize for not stating this clearly. To explain these differences, we traced every CAFE result to its experiment group and compared clean accuracy across groups. All groups use the same ConvNetD3 consumer (100 epochs), so the evaluation network is identical across these tables. The differences come from different groups of synthesis seeds. Each experiment group (the main grid, the hyperparameter sensitivity study, and the compute-matched ablation) re-distilled CAFE independently with its own three synthesis seeds, and each table reports the mean over its own group. Within every group, Vanilla and NOVA share the same seeds and settings, so all paired comparisons remain valid.

Table R7. CAFE clean accuracy (%) on CIFAR-10 IPC10 in different tables.

Experiment groupTable in revised paperVanillaNOVA
Main grid / cross-architectureTables 2, 5, 942.4142.41
Hyperparameter sensitivityTable 23 (λ\lambda=1, ϵ\epsilon=8/255)–42.73
Compute-matched ablationTables 6, 1941.942.0

The resulting clean accuracies differ by less than 1 point, which is comparable to the seed-level standard deviation (0.2–0.5). The revised captions now identify the main-grid, hyperparameter-sensitivity, and compute-matched results as independently distilled three-seed groups; they are not repeated evaluations of the same artifacts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Comments

Dear AC/SAC/PC,

We sincerely thank the reviewers, AC and PC for their efforts during the review process. We would like to provide a summary of our contributions, the reviews, and our responses to facilitate your final assessment.

Our Core Contributions

In this work, we identify a delivery-time vulnerability of dataset distillation. A perturbation added to a distilled set after distillation substantially reduces the accuracy of models trained on it, across all major matching families. To address this, we propose NOVA, a native-objective vicinal alignment framework that extends each method's own matching objective from the delivered set to its neighborhood. NOVA plugs into existing pipelines and leaves the synthetic-set format and downstream training unchanged.

We are encouraged that the reviewers recognized the following strengths:

  • Problem and Motivation: Reviewers found that the paper "identifies a meaningful vulnerability" (Reviewer WyUm), that the robustness of the distilled set as an artifact is "a reasonable thing to point out to the DD community" (Reviewer Lic3), and that it discusses "an important problem" (Reviewer 6uUH).
  • Method Design: Reviewers described NOVA as "simple" (Reviewer Lic3), noted that it uses each backend's existing objective (Reviewer WyUm), and called it "a straightforward framework" (Reviewer 6uUH).
  • Experimental Strength: Reviewers found the experiments "broad" and the improvement "quite consistent" (Reviewer Lic3). They also noted that the evaluation "covers multiple matching families, datasets, and held-out attacks" (Reviewer WyUm).

Summary of Scores

  • Reviewer Lic3: 2. The reviewer noted "If most of the gain remains, I'd be happy to raise my score", which our amplitude matching confirms for most backends. Later, the reviewer noted that our budget, noise and RDC/ROME results "address those concerns". Our follow-up response has not yet received further feedback.
  • Reviewer WyUm: 3. The reviewer found that the paper "identifies a meaningful vulnerability" and is "clear and easy to follow".
  • Reviewer 6uUH: 2. After our responses and clarifications, the reviewer noted "I would like to raise my justification from weak reject to borderline". Our further clarification has not yet received feedback.

Summary of Revision to Address the Raised Concerns

For Reviewer Lic3:

  • Added bidirectional amplitude matching and RMS-capped distillation, and state that amplitude explains much of the CAFE gain (Appendix E.2, F.1 and H.3).
  • Corrected the attack budgets, added budget-controlled attacks and a random-noise control, and clarified the threat model (Sec. 5.1, Appendix E.3 and Limitations).
  • Added native-loss PGD and an attack through downstream training, and described the mixed settings accurately (Sec. 5.3 and Appendix E.4).
  • Clarified the evaluation protocol, corrected MTT, and validated NOVA under official recipes (Appendix D.2).
  • Explained the CAFE differences by independent seed groups, and added a compute-matched Native Ascent Consistency ablation (Sec. 5.6 and Appendix F.1).
  • Added a common-consumer comparison with RDC and ROME (Sec. 5.2 and Appendix E.5).

For Reviewer WyUm:

  • Clarified the research gap relative to Poisoned Distillation, GUARD and ROME, and isolated NOVA's contribution with matched-parent, common-consumer and compute-matched controls (Sec. 1, 5.2, Appendix E.5 and F.1).
  • Corrected the attack budgets, added a controlled training-radius sweep, budget-controlled attacks and a random-noise control (Sec. 5.1, Appendix E.3 and F.2).
  • Added amplitude matching, RMS-capped distillation, and clipping and quantization controls (Appendix E.2, F.1 and G.1).
  • Traced every table to its experiment group, reran CAFE on EuroSAT with both weights, and validated NOVA under stronger recipes (Appendix D.2 and F.2).
  • Added adaptive attacks and a degradation-reduction metric that separates robustness from clean-accuracy gains (Sec. 5.3 and Appendix E.4).
  • Revised overstated claims, reported uncertainty, and added a matched small-subset control (Appendix E.7).

For Reviewer 6uUH:

  • Extended the evaluation to 224x224 ImageNet-100 and to the full ImageNet-1K with four backends, DM, TESLA, SRe2L and CDA, and compared our results with the published ones (Appendix D.2 and H.1).
  • Compared NOVA with consumer-side purification and with dedicated UE defenses, UEraser, PGD-AT and AVATAR, and showed that NOVA can be combined with them (Appendix G.1 and H.2).
  • Added accuracy-versus-budget curves for EMN, LSP, CWN, GUE and FUSE (Sec. 5.3 and Appendix E.3).
  • Reported the distillation cost on larger datasets and added it to the Limitations (Sec. 6 and Appendix F.4).

We sincerely appreciate the reviewers' constructive suggestions, which have made our paper and method more complete. We hope this summary assists with your final assessment. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.