More Information, Less Uncertainty? Monotonic Uncertainty Regularization under Progressive Information

Many real-world predictive systems acquire information progressively. In hospitals, for example, clinicians may order a costlier, slower, or more invasive test only when a simpler one is inconclusive. At each step, a clinician must decide whether she has enough evidence or should acquire more, and predictive uncertainty is a natural signal for this choice. In machine learning, however, such uncertainty is usually evaluated and assessed at each information level individually, not along the sequence of progressive acquisition where these decisions must be made based on uncertainty. Through empirical evaluation, we uncover a phenomenon of aggregate monotonicity illusion: as inputs exhibit richer and cleaner signal, models become more accurate and less uncertain on average across a whole dataset, yet when considering each sample separately, their uncertainty increases with additional signal more than half of the times. This holds across three datasets, Imagenette, ImageNet-1k, and chest X-ray classification, spanning different scales, domains, tasks, and architectures. We address this issue with Monotonic Uncertainty Regularization (MUR), a family of lightweight fine-tuning objectives that links predictions for the same sample across multiple information levels and encourages progressively sharper predictive distributions. MUR significantly reduces the issue of aggregate monotonicity illusion while maintaining predictive performance, and improves the cost–performance trade-off of progressive acquisition. These results caution against relying on aggregate uncertainty trends for per-sample acquisition decisions and show that lightweight cross-level fine-tuning can make uncertainty a more reliable signal for deciding whether to acquire additional information.