acceptodds
Under review as a conference paper at ICLR 2027

Teacher–Student Disagreement For Uncertainty-Aware Human-In-The-Loop AI

Abstract

Large-scale classifiers are vulnerable to adversarial test-time attacks: small, deliberately crafted input perturbations that cause confident misclassification. Defenses such as adversarial training give only empirical robustness, and formal certification does not scale to modern models, so a deployed system still needs a mechanism for deciding when a prediction should be deferred to a human expert. Such selective prediction schemes typically use the model's own confidence, its maximum softmax probability, as the deferral trigger. However, our analysis shows that under attack this score is not merely unreliable but inverted: on CIFAR-10 and MNIST, attacked inputs receive higher confidence than clean ones, so a confidence-based rule defers less exactly when it should defer more. We propose a framework that pairs a standard-trained teacher model with a small ensemble of lightweight student models that are distilled from the teacher, adversarially trained, and uses teacher–student disagreement as a signal that the teacher may have been compromised. Against predictive entropy, mutual information and vote margin computed on the ensemble, and against nine selective-prediction and adversarial-detection baselines, disagreement is the strongest signal for deciding which teacher predictions to trust, that is, for ranking the teacher's wrong predictions ahead of its correct ones so that a human reviews the wrong ones. For instance, it ranks teacher errors with an area under the ROC curve (AUROC) of – where confidence scores –. Adversarial detectors such as Mahalanobis distance detect attacks as well as disagreement does but cannot rank teacher errors. Because attacks crafted for the teacher rarely transfer to the robust students, the ensemble also serves as a fallback predictor (– accuracy where the teacher is at –). The same mechanism sets the limits: on clean inputs confidence remains the better selector; measures computed on the students alone are blind to attacks that do not transfer to them; the students must be accurate enough to serve as a reference, a condition the practitioner can check from clean accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial robustness, selective prediction, learning to defer, human-in-the-loop, teacher–student ensembles, knowledge distillation, adversarial training, uncertainty quantification, ensemble disagreement, adversarial example detection, misclassification detection, confidence calibration, attack transferability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QVYBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper makes three major contributions. First an adversarial detection setup is proposed in the form of a hybrid selective-prediction framework. From my understanding a disagreement between the student and the teacher serves as a detection rule (and the best result). Second empirical results are shown on CIFAR-10, GTSRB and MNIST. Third, an explanation for bounds on the method is given.

Critical strengths

I like that the authors used a wide range of datasets including CIFAR-10, GTSRB and Tiny-ImageNet. Experimentally I think the results are satisfactory.

Critical weaknesses

There are two major issues that I can see with the current work.

Issue #1: Discussion of relevant literature. When I was reading through, I had a feeling I had seen a similar concept proposed in a lot of papers that you have not cited or discussed in relation to your work.

A) Disagreement between models as a form of detection adversarial examples was proposed in the Barrier Zone paper:

https://ieeexplore.ieee.org/abstract/document/9663375

Model disagreement was further used as a detection method for ensembles in a game theoretic based adversarial defense here: https://ieeexplore.ieee.org/document/11155075

B) Architecture diversity as a form of KD adversarial training: https://www.sciencedirect.com/science/article/pii/S1877050925021647

To be clear I am NOT claiming your method is not novel. But I would like to see a discussion and citations for how your method is built on top of what appears to be earlier methods in the literature that are not properly cited.

Issue #2: Experimental problems. I do not want to see FGSM or PGD results in a paper that has AA. FGSM and PGD are older algorithms and it is well established in the literature that APGD (from AA) outperforms it in most cases. I can dig out appropriate citations to back up my claim if needed but really this should just be common knowledge at this point. I suggest the following: report AGPD in each table INSTEAD OF PGD. Move all FGSM and PGD results to the appendix. Still report AA because it is a great combination of attacks and may do even better than APGD in some strange cases.

(minor) Issue #3: AA does include a black-box attack (square), but I would be curious what a true hard label black-box attack results might achieve. E.g. RayS or ADBA:

Rays: https://arxiv.org/abs/2006.12792

ADBA: https://arxiv.org/abs/2406.04998

As a reviewer I CANNOT mandate you do anymore experiments. I think the set of attacks you did is OK. I would just like to mention it would definitely make it easier for me to argue in favor of acceptance with other reviewers if the entire suite (white-box and SOTA black-box attacks were included).

Other minor comment: Is it possible for you to report w-robust for each model configuration to make it very clear which configuration performs best? You can find the w-robust metric used in other top tier papers like this one: https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136640577.pdf

Questions for authors
  1. Is it possible for you to improve the writing by discussion and citing proper foundational works in this domain?

  2. Is it possible for you to improve the experimental discussion by removing non-SOTA white-box attacks and/or reporting w-robust?

Reviewer_8gJWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that pairs a standard-trained teacher model with a small ensemble model composed of a set of lightweight student models distilled from the teacher model. These student models undergo adversarial training, and the prediction discrepancies between the teacher and student models are used as signals indicating that the teacher model may have been attacked. The discrepancies between the teacher and student models can prioritize the handling of incorrect predictions by human review.

Critical strengths
  1. This paper not only examines whether the method can identify the attacked samples, but also checks whether it can identify the samples where the teacher model made mistakes. This distinction is very necessary because methods like Mahalanobis may be very good at detecting attacks, but they may not be reliable when it comes to determining which samples the teacher model will make mistakes on. The author evaluated these two matters separately, and the result is more persuasive and more relevant to the practical question of "whether to seek human assistance or not".
  2. In Table 2, the number of human queries was consistent. This made the comparison of accuracy more persuasive. Otherwise, if almost all samples were given to humans, the accuracy would naturally increase. The paper also tested different budget settings. The ablation experiments also have reference value. When adversarial training was removed, the performance of the method in adaptive attacks significantly decreased, which indicates that adversarial training indeed played an important role.
  3. The author candidly pointed out the limitations of the study, including false positives in the clean sample, insufficient performance of the student model on Tiny ImageNet, the use of only one random seed in the experiments, the assumption that human judgment is always correct, and the need for further improvement in the assessment of adaptive attacks.
Critical weaknesses
  1. The experimental results do not match the description of the strongest robustness. Appendix H adjusted the KL weights for one joint cross-entropy/KL attacks on GTRSB. However, this does not prove convergence, nor does it rule out the possibility that there is a better objective resulting in an acceptable error.
  2. The students are already quite excellent, yet the teachers were still retained. In the GTSRB PGD, the student can achieve an accuracy rate of 85.2% without the need for queries. However, the teacher has only an accuracy rate of 58.8% under a 50% query budget. There is a lack of a comparison with the same query budget.
  3. The new contributions of this paper compared to existing methods are not yet clear. What is presented at present is mainly a combination of existing methods. Bagnall et al., Training Ensembles to Detect Adversarial Examples (2017), use ensemble rejection and suggest a separate stronger classifier for accepted inputs as future work. The teacher–student comparison merits investigation, but needs evidence isolating its added value. Likewise, the clean-trained logistic selector is no substitute for a comparison with robust deferral methods such as SARD (Montreuil et al., 2025).
Questions for authors
  1. Under attacks aimed at a common incorrect class, how often is an incorrect teacher prediction accepted? Please report accepted-error frequency, query rate, and selective risk after increasing the steps and restarts and selecting successful attack candidates.
  2. Please compare teacher deferral, student-ensemble deferral, and switching between their predictions on mixed streams. With validation-calibrated policies and one equal total human-query budget, what changes as attack prevalence rises?
  3. How are ties in the binary disagreement score resolved for fixed-budget oracle accuracy and AURC? Are results averaged over random tie-breaking?
  4. What establishes the suggested 20-percentage-point clean-accuracy gap and the need for distillation? Also, 50.0% is not the random-deferral result on GTSRB PGD. At a 50% query budget, that baseline is 54.4%, as Table 2 correctly reports.
Reviewer_8Hx7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates selective prediction under adversarial attacks, where a model’s confidence may become misleading. It pairs a standard-trained teacher with six lightweight students trained through knowledge distillation and adversarial training. Disagreement between the teacher and the student majority determines whether a prediction should be deferred to a human oracle. Experiments on MNIST, CIFAR-10, GTSRB, and Tiny ImageNet compare disagreement with confidence scores, ensemble uncertainty measures, and adversarial detectors. The paper distinguishes attack detection from teacher-error detection and reports strong error-ranking performance, particularly under attacks targeting the teacher.

Critical strengths
  1. A useful distinction between detecting attacks and identifying incorrect predictions. The paper evaluates whether a score prioritizes actual teacher errors rather than merely separating attacked from clean inputs. This distinction matters when human review capacity is limited. On GTSRB under PGD, disagreement achieves teacher-error AUROC of 96.4, compared with 10.1 for low confidence and 47.1 for Mahalanobis distance, demonstrating a substantial advantage in this setting.

  2. The analysis acknowledges important limitations and examines the mechanism. The paper reports the clean-data disadvantage, the failure with weak Tiny ImageNet students, an ensemble-aware adaptive attack, and ablations of distillation, adversarial training, and architectural diversity. These results help identify when disagreement is informative and show that adversarial training contributes more than architectural diversity.

Critical weaknesses
  1. The adaptive evaluation is insufficient to establish the reliability of the complete deferral system. Most attacks target only the teacher, with students affected through transfer. Under this threat model, the advantage of comparing an attacked teacher with comparatively unaffected robust students is understandable, but it does not establish robustness against an adversary targeting the deployed system. The paper includes an adaptive attack, which is valuable, but evaluates primarily one joint-loss formulation with 20-step PGD and a trade-off-weight sweep on GTSRB. It does not sufficiently test attacks directly optimizing the relevant failure event: an incorrect teacher prediction that is accepted rather than deferred, for example by inducing agreement on a common incorrect class. Section 5 acknowledges this limitation. Because adversarial reliability motivates the method, this gap materially limits the conclusions.

  2. The comparative evidence does not sufficiently establish the added value of the proposed ensemble design. The framework combines established components—distillation, adversarial training, ensembling, and disagreement-based deferral—but does not establish whether six students are necessary or preferable to a simpler robust reference. The same-architecture ensemble in Table 5 performs similarly to the diverse ensemble, suggesting that the architectural design is not a major contributor. Comparisons with a single robust student and a reference model under comparable computational budgets would clarify whether the proposed construction provides a meaningful advantage. The reported single-seed results further limit confidence in the smaller differences between configurations. Consequently, the empirical observation is interesting, but the evidence for the specific methodological contribution remains incomplete.

Questions for authors
  1. How does the system perform against attacks explicitly targeting accepted teacher errors, including targeted agreement on an incorrect class? Do the conclusions remain stable with more optimization steps and multiple restarts?

  2. How does disagreement with one robust distilled student compare with disagreement with six students? Can you provide a comparison with a single reference model under a comparable parameter or inference-compute budget?

  3. Since S2 assigns only two scores, how are ties handled when computing AURC and selecting exactly 25% or 50% of inputs for human review? Are the near-perfect-budget results in Table 2 stable under randomized tie-breaking, independent of dataset ordering?

1 comment

We sincerely thank the reviewer for their constructive feedback. We have provided additional experimental evidence addressing the main concerns. We hope these additions help clarify the contribution and its limitations. We would greatly appreciate any further feedback, particularly on whether our new experiments and explanations adequately address your concerns. We are happy to clarify any remaining questions.

Comments on the paper

2 comments

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.

Best, AC

AuthorsOpenReview ↗

Summary for the Area Chair

We sincerely thank the Area Chair and all three reviewers for their constructive feedback. We have made a substantial effort to address the principal concerns through new experiments, expanded comparisons, and clearer positioning of our contribution.

Our key additions are:

  1. Stronger adversarial evaluation: We evaluated system-aware adaptive attacks with up to 200 optimization steps and 10 restarts, standardized APGD-CE/APGD-DLR attacks, and hard-label RayS attacks. Under teacher-targeted standardized attacks, S2 achieves approximately 98% teacher-error AUROC on GTSRB. We also explicitly quantify its limitations under adaptive attacks targeting incorrect teacher–student agreement.

  2. System-level effectiveness: We now report accepted-error rates, selective risk, human-review costs, and budget-matched random baselines. Even under the evaluated adaptive attacks, S2 improves oracle accuracy over random review by 2.90 percentage points on GTSRB and 0.67 points on CIFAR-10 at a 50% review budget.

  3. Additional baselines and ablations: We evaluated single versus six robust students, parameter-comparable wider students, architectural diversity, distillation, adversarial training, and student-only deployment. These comparisons clarify when the proposed mechanism is beneficial and when alternative policies perform better.

  4. Clarified novelty and reproducibility: We strengthened the discussion of related work, explicitly acknowledged existing disagreement-based defenses, and examined the stability of our original tie-breaking results.

We emphasize that our contribution is an empirical investigation of how robust reference models can help identify errors made by an unchanged teacher and prioritize limited human-review capacity. The additional experiments strengthen this central contribution.

We respectfully hope that the reviewers and Area Chair will consider these new findings and clarifications in their final assessment. We remain happy to address any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.