acceptodds
Withdrawn from ICLR 2027

Master-and-Stop Training for Out-of-Distribution Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Out-of-distribution (OOD) detection aims to ensure AI system reliability by detecting inputs outside the training distribution. Recent work shows that overfitting during later stages of training can hurt OOD detection. To overcome overfitting, several methods attempt to distill the model after training or prune the model during training from a model-centric perspective. In contrast, this paper proposes a data-centric end-to-end solution called Master-and-Stop Training (MST), which follows the principle that once the model has mastered an instance, training on it should stop to prevent overfitting. MST considers an instance mastered if the zero-order and second-order differences of its uncertainty value remain within a small range around zero, offering a more consistent measure of an instance’s learning status. Additionally, since different classes exhibit varying optimization progress, using a fixed threshold to determine when to exclude an instance from backpropagation is theoretically unsound. MST develops an adaptive threshold by incorporating class-informed statistics to determine when to exclude an instance. Extensive experiments demonstrate that MST can enhance OOD detection performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
out-of-distribution detection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5KhROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Master-and-Stop Training, a data-centric training strategy designed to mitigate the deterioration of OOD detection performance caused by overfitting during the later stages of training. The key idea is to identify samples that have been sufficiently learned and temporarily exclude them from backpropagation. The proposed mastery criterion combines sample-level prediction uncertainty with its second-order temporal difference.

Critical strengths
  1. This method is direct. It modifies sample contributions during training without requiring additional OOD samples or changing the inference-time scoring function.
  2. Rather than designing another post-hoc scoring function, this paper investigates how sample-level training dynamics influence the intrinsic OOD detection capability of a classifier.
Critical weaknesses
  1. Although the experimental results suggest that MST improves OOD detection, it remains unclear whether the improvement specifically comes from identifying mastered samples or from reducing the effective amount of training on easy examples. There should be more ablation study to demonstrate it.
  2. The most important issue: The benchmark and evaluation paradigm are quite old-school. Please consider more modern setting beyond the classification task, which is extensively studied several years ago. Moreover, the selected classification tasks in your main exp are also quite naive!
  3. The paper is low-formatting. i. Contribution is short but unclear; ii. the methodology part don't have a clear overview to present your holistic achitecture; iii. The presentation is verbose in introduction.
  4. The performance looks incremental and trivial, where the performance gain with the help of your method even is negative, c.f., Table 10.
Questions for authors

See weaknesses.

Reviewer_Ks1oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Master-and-Stop Training, a data-centric training strategy designed to improve OOD detection by preventing excessive optimization on already-learned training instances. The main idea is to identify mastered samples based on their uncertainty values and second-order temporal differences, and then exclude these samples from backpropagation. To accommodate heterogeneous learning dynamics across classes, the method further introduces class-adaptive mastery thresholds derived from class-wise uncertainty statistics. The proposed framework is evaluated on CIFAR-10, CIFAR-100, ImageNet, and MedMNIST benchmarks using multiple OOD scoring methods. The results demonstrate substantial improvements.

Critical strengths
  • The proposed framework offers a simple and practical strategy for improving OOD detection without modifying the model architecture or requiring additional OOD data. Its substantial improvements on CIFAR-10, particularly in reducing FPR95 across several scoring functions, demonstrate its potential practical value.

  • The paper evaluates the method across multiple benchmarks and OOD detection methods, including comparisons with relevant sample-selection approaches. The extensive ablation studies further support the empirical usefulness of the proposed framework.

Critical weaknesses
  • The paper defines uncertainty as the negative maximum softmax probability, while the mastery criterion requires this value to be smaller than a positive threshold. Since MSP is always negative, this condition might be automatically satisfied, making the zero-order criterion ineffective. This inconsistency directly challenges the correctness and reproducibility of the method, particularly the ablation results. It would be great to clarify the actual implementation.

  • The assumption that small second-order uncertainty differences reliably indicate sample mastery is not sufficiently justified. The differences do not necessarily imply convergence and can amplify noise. Also, it remains unclear whether the improvements arise specifically from the proposed mastery criterion or from general sample filtering and altered optimization dynamics. It would be great to show stronger controlled evidence. Currently, the contribution and its claimed advantages remain insufficiently verified.

Questions for authors
  • What is the exact uncertainty definition used in the implementation?

  • Can the authors clarify the implementation details and hyperparameter selection procedure? When mastered samples are masked, is the loss normalized by the original batch size or the number of active samples? How are the first two epochs handled when insufficient uncertainty history is available? Are uncertainty values computed under training or evaluation mode, and with which data augmentations?

  • Can the authors compare the method with random masking, simple uncertainty-based filtering, and conventional early stopping while matching the number of active samples, class-wise selection ratios, and computational budgets?

  • Can the authors directly quantify changes in inter-class uncertainty variance and class-wise OOD detection performance? How does the proposed adaptive strategy compare with a global threshold calibrated to achieve the same average sample-retention ratio?

Reviewer_thN7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Master-and-Stop Training (MST), which estimates each training example’s uncertainty and its second-order difference over time. A sample is excluded from backpropagation for the current epoch when both quantities fall below thresholds. The paper also proposes class-adaptive thresholds based on within-class uncertainty variance, and evaluates MST with multiple OOD scoring functions on CIFAR, ImageNet, and MedMNIST.

Critical strengths

The problem is relevant, the method is simple to integrate with existing OOD scores, and the evaluation covers several datasets and baselines.

Critical weaknesses
  1. Criterion validity: Low, stable uncertainty does not necessarily mean a sample is correctly learned; a model may be confidently wrong. The paper does not sufficiently justify why a small second-order difference indicates mastery.

  2. Adaptive thresholds: The variance normalization is undefined when class variances are equal. Estimation details and the rationale for the threshold direction also need clarification.

  3. Reproducibility and controls: Key implementation and tuning details are missing, including how uncertainty histories are computed and whether OOD data influence threshold selection. Comparisons at matched compute or update budgets are needed.

  4. Strength of claims: Results are not uniformly improved, and the evidence that MST mitigates overfitting is indirect.

Questions for authors

1.Could the authors provide full MST pseudocode and explain how uncertainty histories, data augmentation, BatchNorm statistics, and class-wise variance are handled?

2.How is the zero-denominator case in Equation (6) handled? How are class-wise variances estimated and smoothed, and can the resulting thresholds become negative?

3.Are any OOD validation or test data used to select δ1\delta_1, δ2\delta_2, or γ\gamma? Please describe the data-isolation protocol.

4.Can the authors compare MST with random subsampling, loss-dynamics selection, and a fixed-threshold variant at matched effective update counts or compute budgets?

5.Can the authors provide more direct evidence that MST mitigates overfitting, rather than merely changing training dynamics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.