acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Decision-Boundary Probing for Robust Distillation

Abstract

Robust distillation has been shown to effectively transfer robustness from larger adversarially trained models to smaller student models. However, robust teacher models may not reliably classify all adversarial examples, and conventional distillation mostly do not explicitly account for the position of these examples relative to the teacher's decision boundary. In this work, we propose Adaptive Decision-Boundary Probing Distillation (ADBD), a robust distillation framework that uses the teacher's classification behavior to construct sample-specific query points. Starting from an adversarial example generated through coordinated teacher-student information, we preserve its perturbation direction while adaptively scaling its magnitude. When the teacher misclassifies the adversarial example, the perturbation magnitude is reduced to obtain a query point closer to the teacher's decision boundary while remaining on the teacher-correct side. When the teacher correctly classifies the adversarial example, the perturbation is extended along the same direction to probe a farther region in which the teacher remains correct. The resulting query samples provide teacher supervision at locations selected according to the teacher's local decision-boundary behavior. We evaluate ADBD across multiple datasets and adversarial attacks. Our experimental results show that ADBD improves the adversarial robustness of distilled student models over existing robust distillation methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Robust Distillation, Adversarial Robustness, Adversarial Attacks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZWjtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Adaptive Decision-Boundary Probing Distillation (ADBD). Starting from an adversarial example generated using coordinated teacher-student information, ADBD rescales the perturbation along the same direction. If the teacher misclassifies the adversarial point, the method moves toward the clean input to find a teacher-correct point near the boundary; if the teacher remains correct, it extends the perturbation to probe a farther correct region. These query points provide supervision about the teacher's local decision behavior. Experiments on CIFAR-10, CIFAR-100, and SVHN report improvements over several robust-distillation baselines under PGD, C&W, and AutoAttack.

Critical strengths
  1. The motivation is clear: a robust teacher is not uniformly reliable on student-generated adversarial examples, so supervision should depend on the teacher's sample-specific behavior.

  2. The evaluation includes two public robust teachers, two student architectures, three datasets, several attacks, and comparisons with ARD, IAD, RSLAD, AKD, AdaAD, and SmaraAD.

  3. ADBD consistently improves robust accuracy in most configurations. For example, it reaches approximately 54.8% AutoAttack accuracy in a CIFAR-10 ResNet-18 setting. Three-seed results and objective ablations support the contribution of query-point supervision.

Critical weaknesses

ADBD introduces additional teacher evaluations and per-example bisection, but the paper does not systematically report wall-clock time, teacher-forward counts, memory use, or computational overhead relative to the baselines.

Questions for authors

1.Based on existing training logs, how many additional teacher forward passes and how much training time does ADBD require relative to AdaAD or SmaraAD? A large cost would lower my assessment, whereas modest overhead would strengthen the practical contribution.

  1. I am curious about the comparison between the proposed method and existing dual-teacher robustness distillation.
Reviewer_AcXbOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Decision Boundary Distillation (ADBD), a framework that transfers adversarial robustness from a strong teacher model to a smaller student model. The method first generates adversarial examples by maximizing the KL divergence between the student’s and teacher’s predictions. Then, according to the teacher’s classification behavior, it adaptively adjusts the magnitude of perturbations along the original adversarial direction.

The method was evaluated on the simple datasets CIFAR-10, CIFAR-100, and SVHN, using WideResNet as the teacher model and ResNet-18 or MobileNetV2 CNN architectures as the student models. The reported results show a slight improvement in adversarial robustness compared to selected baselines under PGD, C&W loss-based attacks, and AutoAttack.

Critical strengths
  1. The paper was evaluated on three datasets with different teacher and student models. In the reported PGD, C&W, and AutoAttack results, ADBD generally performed better than the chosen baselines, suggesting that this method has some practical application potential.

  2. It's fairly reasonable to adaptively create query samples for knowledge distillation based on how the teacher model performs on different samples. This method also takes into account how to handle samples when the teacher model makes incorrect predictions.

Critical weaknesses
  1. The method proposed in this paper seems to be mainly a combination of existing techniques rather than a substantial algorithmic innovation. Specifically, in Eq. 6, the inner adversarial optimization directly follows AdaAD[1], and in Sec 4.1, adjusting the perturbation magnitude in both directions based on classification results and using binary search to find decision boundaries is very similar to the AN-PGD algorithm in MMA[2]. Although the authors acknowledge the similarity with MMA and emphasize that the two have different optimization goals, this difference mainly lies in how the information from the boundary search is used, rather than the boundary search mechanism itself. So, it’s still unclear what substantial new algorithmic mechanism ADBD introduces beyond applying existing techniques to robust knowledge distillation. It would be great if the authors could clarify this in more detail.

  2. The authors set the maximum perturbation scaling factor to αmax=2\alpha_{max}=2, while the perturbation budget for the original adversarial sample is ϵ=8/255\epsilon = 8/255. From the provided code, it appears that query samples can exceed the original perturbation budget without being reprojected into the original perturbation constraint. Therefore, it remains unclear whether the observed robustness improvement comes from decision boundary-guided query sample selection or additional data enhancement from a larger perturbation scope. It is hoped that the authors can supplement the relevant ablation experiments and explain the rationale for this design.

  3. Although the proposed method achieves relatively robust results across various experimental settings, the improvement doesn't seem very significant. Considering that the method introduces additional costs for teacher model queries and decision boundary searches, and lacks sufficient ablation studies on the core mechanism, it's still unclear whether these relatively limited performance gains are enough to justify the contribution of the proposed method. It would be great to provide some related explanations.

[1] Boosting Accuracy and Robustness of Student Models via Adaptive Adversarial Distillation.

[2] Max-Margin Adversarial (MMA) Training: Direct Input Space Margin Maximization through Adversarial Training.

Questions for authors
  1. Based on the issues mentioned above, I hope the author can elaborate more on the substantial differences between ADBD and AdaAD, MMA. Specifically, apart from applying existing adversarial optimization and decision boundary search techniques to robust knowledge distillation, what new algorithmic mechanisms does ADBD actually introduce?

  2. It would be great if the authors could add ablation experiments for fixed perturbation scaling, random perturbation scaling, and without using decision boundary search to verify the independent contribution of the proposed mechanism. Also, it seems the code allows queries to exceed the original ϵ=8/255\epsilon=8/255 perturbation budget. Could the authors explain this design, and perhaps run control experiments while keeping the perturbation budget in check or matching the actual perturbation strength to rule out the effect of stronger data augmentation?

  3. I hope the author can explain the parameter settings of each baseline in detail, add some experiments with recent related methods, and consider how the method can generalize under the VIT architecture, as well as its capabilities on more complex datasets.

  4. Also, I want to know what the computational cost of this method is. After all, it goes through multiple optimization steps, but the results seem almost the same as the baseline, so I’m really curious about your computational cost.

Reviewer_fCrXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Adversarial example evasions attacks are an important field of AI security. Knowledge distillation is the process of training less complex models based on the outputs from larger trained models. This paper considers the combination of the two fields, the distillation of robustness to less complex models. A new method for robust knowledge distillation, Adaptive Decision-Boundary Probing Distillation (ADBD) is proposed. The main innovation in the method is the use of query examples based on the teacher’s classification behavior. Experimentally results are shown on CIFAR-10, CIFAR-100 and SVHN.

Critical strengths

The paper writing is adequate.

The experimental results demonstrate that the method does perform better in some cases.

Critical weaknesses

I think there are two main issues with the paper in the form of citations/discussion of relevant work and in the experimental results/presentation.

Issue #1: Discussion of relevant literature. I was not able to completely understand/check all of the mathematics of your paper (limitation on my part, not yours) but when I was reading through I had a feeling I had seen a similar idea proposed in the paper Friendly Adversarial Training (FAT):

https://arxiv.org/pdf/2002.11242

While FAT is NOT a knowledge distillation technique, it is an adversarial training technique and if I understand correctly, does bear some semblance to your algorithm in the form of choosing HOW much to perturb the adversarial example based on whether it is classified correctly or not. I am not saying your method is not novel, but I am saying you should try and explain how your work fits in to the larger scheme or maybe is similar/dissimilar to FAT with a proper citation/discussion.

Likewise I can think of few other KD adversarial distillation works that the authors do not cite or discuss which I think could be cited and give important context to their work:

A) DARHT one recent KD adversarial training method that outperforms RSLAD in some cases: https://www.sciencedirect.com/science/article/pii/S1877050925021647

B) MTARD another recent KD adversarial training method: https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136640577.pdf

Issue #2: Experimental problems. I do not want to see PGD results in a paper that has AA. PGD is an older algorithm and it is well established in the literature that APGD (from AA) outperforms it in most cases. I can dig out appropriate citations to back up my claim if needed but really this should just be common knowledge at this point. I suggest the following: report AGPD in each table INSTEAD OF PGD. Move all PGD results to the appendix. Still report AA because it is a great combination of attacks and may do even better than APGD in some strange cases.

Finally, I would like to see one other metric reported. I will preface this by saying it is often hard to compare robustness and clean accuracy at the same time across multiple models and attacks. Please report w-robust (the average of clean accuracy and the worst case attack robustness) for each model. You can see papers like MTARD above and DARHT which do exactly that. If you don’t have the best w-robust, I don’t see the point in your method.

Minor comment:

=A lot of reviewers usually ask for additional datasets like ImageNet. I realize that might not be computationally possible but would it be possible to do say Tiny-ImageNet? I am NOT dropping my score based on this, I am just hinting that it would further strength your claims a ton to have a slightly larger dataset (as both C10/C100 have been done to death at this point).

Questions for authors
  1. Can you include citations to the literature/better discussions of related algorithms like FAT?
  2. Can you fix your results so that we don’t need to look at ugly old PGD anywhere in the main paper? Specifically, we need to see APGD and AA, and the combined w-robust measurement to have a better understanding of the results.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.