acceptodds
Under review as a conference paper at ICLR 2027

More Information, Less Uncertainty? Monotonic Uncertainty Regularization under Progressive Information

Abstract

Many real-world predictive systems acquire information progressively. In hospitals, for example, clinicians may order a costlier, slower, or more invasive test only when a simpler one is inconclusive. At each step, a clinician must decide whether she has enough evidence or should acquire more, and predictive uncertainty is a natural signal for this choice. In machine learning, however, such uncertainty is usually evaluated and assessed at each information level individually, not along the sequence of progressive acquisition where these decisions must be made based on uncertainty. Through empirical evaluation, we uncover a phenomenon of aggregate monotonicity illusion: as inputs exhibit richer and cleaner signal, models become more accurate and less uncertain on average across a whole dataset, yet when considering each sample separately, their uncertainty increases with additional signal more than half of the times. This holds across three datasets, Imagenette, ImageNet-1k, and chest X-ray classification, spanning different scales, domains, tasks, and architectures. We address this issue with Monotonic Uncertainty Regularization (MUR), a family of lightweight fine-tuning objectives that links predictions for the same sample across multiple information levels and encourages progressively sharper predictive distributions. MUR significantly reduces the issue of aggregate monotonicity illusion while maintaining predictive performance, and improves the cost–performance trade-off of progressive acquisition. These results caution against relying on aggregate uncertainty trends for per-sample acquisition decisions and show that lightweight cross-level fine-tuning can make uncertainty a more reliable signal for deciding whether to acquire additional information.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
regularization, monotonic uncertainty, progressive information acquisition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HRGaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether predictive uncertainty can reliably guide a system in acquiring more informative observations for the same sample. It aims to improve the cost–performance trade-off by encouraging sample-level uncertainty to decrease as the information level increases. MUR-U and MUR-AU formulate training objectives based on entropy ordering and cross-level prediction distribution matching, respectively. Overall, the framework is clearly presented, and the experiments cover a reasonably broad range of settings. However, some numerical inconsistencies remain, and the current experimental design is insufficient to validate the paper's central claims.

Critical strengths

(1)The paper focuses on sample-level decisions, where average entropy may decrease even as uncertainty increases for individual samples. It uses the violation rate VRVR to measure this inconsistency and links it to the stopping policy. (2)Experiments across five random seeds evaluate the methods on Imagenette, ImageNet-1k, CXR-3, and CXR-14, covering multiclass and multilabel tasks, two network architectures, and different types of information. (3)The two methods have distinct effects. MUR-U enforces entropy ordering, while MUR-AU also transfers class distributions across information levels. On Imagenette, MUR-U achieves a violation rate of only 0.8%, yet its routing performance is 5.9% lower than that of cross-entropy training, showing that a lower violation rate does not necessarily lead to better routing. (4)Temperature scaling yields violation rates close to those of standard training, while Gatekeeper fails to maintain both low-information performance and routing gains. An additional tolerance analysis examines whether the violations result from negligible entropy fluctuations.

Critical weaknesses

(1) The abstract states that uncertainty increases for more than half of the samples when additional signals are provided. However, the reported proportion for CXR-3 is only 49.8%. In Fig. 3a, the transition-level and sample-level proportions are 26.0% and 73.5% for ImageNet-1k, and 37.1% and 90.4% for Imagenette, respectively. These two types of statistics should be clearly distinguished, and the statement in the abstract should be revised accordingly. (2) Each CXR-14 sample has only one transition between information levels. Therefore, the sum of the violation rates across groups should equal the sample-level violation rate. However, the group-level proportions in Fig. 9 sum to 51.0%, whereas Fig. 12 reports a violation rate of 54.2% under BCE. For CXR-3, the group-level proportions sum to 49.4%, which differs slightly from the reported value of 49.8%. (3) The paper claims that both MUR methods eliminate violations on CXR-3 and CXR-14. However, Appendix D reports residual violation rates of 0.1%–3.3% across CXR-3 disease categories, while Effusion and Infiltration on CXR-14 also retain violations. The authors should verify these results and avoid claims that are inconsistent with the reported evidence.

Questions for authors

(1) Equation (4) reflects a general property of averages. A decrease in average entropy does not guarantee that individual samples become less uncertain as more information is provided. Therefore, the paper's central empirical claim should be supported by the measured violation rates rather than inferred from average entropy. (2) The paper acknowledges that richer observations may increase uncertainty for some samples. However, Eqs. (9) and (10), together with the model selection rule based on the lowest validation violation rate, penalize each increase in entropy, allowing a tolerance of five percentage points only at the most informative level. Moreover, the performance gains of MUR-AU and MUR-U are only approximately 1.3% and 1.4%, respectively. The authors should further analyze the relationship between uncertainty constraints and actual routing gains, and clarify whether these constraints lead to consistent performance improvements. (3) Appendix A.2 uses 10 training epochs and 10 fine-tuning epochs for CXR, which does not support the claim that fine-tuning takes approximately half as long as training on CXR. In addition, the x-axis of Fig. 4 ends at 1 for CXR-3, while the axis range for natural images may also be inconsistent with the actual cost definition. In contrast, Fig. 13 extends the x-axis to 1.25 for CXR-14. The cost definition should be unified, and the corresponding figures and claims should be rechecked. (4) Eqs. (12) and (13) assign unit weights to the adjacent-level distribution-matching terms in MUR-AU, without including the factor used in the task loss or a penalty weight corresponding to λ\lambda in MUR-U. The authors should consider adding a baseline that improves predictions at lower information levels without explicitly enforcing uncertainty ordering. This would help distinguish gains from improved predictive performance from those attributable to the uncertainty monotonicity constraint.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HRGa (1/2)

Dear Reviewer HRGa,

Thank you for the careful and constructive review. We are encouraged that you found the sample-level focus well motivated, the evaluation broad (five seeds, multiclass and multilabel tasks, two architectures, two information types), and the distinct effects of MUR-U and MUR-AU informative. Your detailed reading helped us make the paper more precise: we clarify measures that we had not distinguished clearly, make several statements more precise, and add baselines that separate better predictions from uncertainty ordering. All results are means over five seeds. We address each point below.

[W1] The abstract states that uncertainty increases for more than half of the samples, but CXR-3 is 49.8%. Transition-level and sample-level proportions should be clearly distinguished, and the abstract revised accordingly.

Thank you for pointing this out. The two rates measure different things, which we did not distinguish clearly: the sample-level rate counts samples whose uncertainty increases at any step, while the transition-level rate counts individual steps.

CE/BCE violation rate (%)ImagenetteImageNet-1kCXR-3CXR-14
Sample-level90.473.549.854.2
Transition-level37.126.049.854.2

CXR has a single transition, so the two coincide.

We will define and report both throughout. Since CXR-3 (49.8%) is just below half, we will also make the abstract precise: uncertainty increases for about half or more of the samples (49.8–90.4% across settings).

[W2] Each CXR-14 sample has one transition, so the group-level proportions in Fig. 9 (51.0%) should sum to the sample-level rate in Fig. 12 (54.2%). For CXR-3, 49.4% vs. 49.8%.

Thank you for checking this carefully. The numbers are consistent, but we did not state their units: Fig. 12 counts one violation per sample (entropy averaged over findings), while Fig. 9 groups (sample, finding) pairs.

Violation rate (%)Per sample (Fig. 12)Per (sample, finding) (Fig. 9)
CXR-349.849.4
CXR-1454.251.0

We will state the unit in both captions and explain the difference in the text.

[W3] The claim that both MUR methods eliminate violations on CXR-3 and CXR-14 is inconsistent with residual per-finding violations in Appendix D (0.1–3.3% on CXR-3; Effusion and Infiltration on CXR-14).

Thank you; we agree that "eliminate" should be stated more precisely. MUR eliminates violations at the sample level (0.0%), while residuals remain for individual findings, mainly Effusion and Infiltration on CXR-14:

Per-finding violation rate (%)BCEMUR-U (corrected)MUR-AU
CXR-3 (3 findings)43–540.0–1.00.2–3.1
CXR-14 (14 findings)43–600.0–13.40.0–11.5

Highest residuals on CXR-14: Effusion (MUR-U 13.2%, MUR-AU 11.5%) and Infiltration (MUR-U 13.4%, MUR-AU 8.4%).

We will revise the claim to: "MUR eliminates sample-level violations on CXR-3 and CXR-14 and reduces per-finding violations from 43–60% to at most 13.4%", and report per-finding results in the main text.

[Q1] Eq. (4) reflects a general property of averages. The central empirical claim should be supported by measured violation rates rather than inferred from average entropy.

We agree, and this is precisely the role of Eq. (4): it shows that aggregate trends cannot establish per-sample monotonicity, which is why all our claims rest on measured violation rates (Fig. 2; [W1]). We will make this explicit around Eq. (4).

AuthorsOpenReview ↗

Response to Reviewer HRGa (2/2)

[Q2] Eqs. (9)–(10) and the selection rule penalize every entropy increase, although richer inputs can legitimately increase uncertainty for some samples. The gains of MUR-AU and MUR-U are only about 1.3% and 1.4%. Analyze the relationship between the uncertainty constraints and routing gains, and whether they lead to consistent improvements.

Thank you for this important question.

Penalizing every increase. MUR is a soft regularizer, not a hard constraint: λ (MUR-U) and α (MUR-AU) trade ordering against accuracy, so increases are discouraged rather than forbidden, and some remain (e.g., 12.3% for MUR-AU on ImageNet-1k). Two observations suggest that frequent CE/BCE increases are not only due to legitimate ambiguity: in our clinician pilot, the radiologist lost confidence on the clean image in only 2 of 30 selected cases (General Response, [2]), and the increases persist when small entropy changes are ignored (App. B.1). We will state explicitly that MUR targets systematic, not universal, monotonicity.

Size and consistency of the gains. The gains of about 1.3–1.4% are on CXR, where routing differences are within one standard deviation; on natural images they are larger:

HV gain over CE/BCE (%)ImagenetteImageNet-1kCXR-3CXR-14
MUR-U (corrected)+2.1+13.0−1.1+0.3
MUR-AU+6.9+18.9+1.3+1.7

MUR-AU therefore reduces violations in every setting without lowering routing in any.

Relationship between ordering and routing. Ordering makes the routing signal more consistent: under CE/BCE and both baselines, about two thirds of the acquisition cost on natural images is spent after the routing confidence has dropped, i.e., after an acquisition left the model less certain. MUR reduces this to 4–38% (see our reply to yo5t), so acquisition decisions rely less on a signal that reverses as information is added. Routing gains also depend on accuracy at each level; e.g., MUR-U has fewer violations than MUR-AU on Imagenette but a smaller gain, so we report both.

[Q3] Appendix A.2 uses 10 training and 10 fine-tuning epochs for CXR, which does not support the claim that fine-tuning takes about half as long. The x-axis of Fig. 4 ends at 1 for CXR-3, whereas Fig. 13 extends to 1.25 for CXR-14; the cost definition should be unified.

Thank you for catching both.

Fine-tuning time. You are right that the claim holds only for ImageNet-1k; we will correct App. A.2:

EpochsImagenetteImageNet-1kCXR-3CXR-14
CE/BCE training20101010
MUR fine-tuning2051010

Cost axes. All figures use the same cost definition (App. A.3); for CXR, the exit costs are 0.25 and 1.25 in both datasets. In Fig. 4, the points at 1.25 are plotted, but the last tick label shown is 1.00. We will add this tick and unify tick labels across figures.

[Q4] Eqs. (12)–(13) assign unit weights to the distribution-matching terms in MUR-AU, without a weight corresponding to λ. Consider a baseline that improves lower-level predictions without explicitly enforcing uncertainty ordering.

Thank you for this suggestion. In MUR-U, λ weights the ordering penalty. In MUR-AU, α plays this role instead: it sets the softness of the distillation target, and thus how strongly ordering is encouraged. A separate weight is therefore unnecessary, so we fix it to 1 and tune only α; we will clarify this in Eqs. (12)–(13). At α = 1, MUR-AU reduces to plain distillation, which is exactly the baseline you suggest. We added it, together with continued CE/BCE training for the same number of epochs:

Violations (%) / HV gain (%)ImagenetteImageNet-1kCXR-3CXR-14
CE/BCE90.4 / 073.5 / 049.8 / 054.2 / 0
Continued CE/BCE92.1 / −3.178.4 / +17.560.3 / +0.959.4 / +1.8
Distillation (α = 1)77.8 / −1.074.4 / +18.948.6 / +1.758.5 / +2.1
MUR-AU4.7 / +6.912.3 / +18.90.0 / +1.30.0 / +1.7

Both baselines improve poorer-level predictions, and on ImageNet-1k also routing, but violations remain at 49–92%. On ImageNet-1k, distillation reaches the same +18.9% routing gain as MUR-AU, but with 74.4% vs. 12.3% violations. Improvements in predictive performance and routing therefore do not automatically yield ordered uncertainty, which MUR explicitly encourages.

Thank you again for your time and careful reading. We will incorporate all changes into the revised PDF, which we will upload by 15 November. Your feedback has made the paper more precise, and we believe the clarifications and additional analyses substantially strengthen the work. We hope you find our response satisfactory and will consider raising your score.

Thank you,

Authors

The authors responded better than I expected. My main concerns (W5/W7) are answered with new control experiments, so I have raised my score to 3. I still ask them to state in the paper what their own numbers show, that the routing gains do not mainly come from the ordering constraint.

AuthorsOpenReview ↗

Response to Reviewer HRGa: Thank you for your feedback and for raising your score

Dear Reviewer HRGa,

Thank you again for your time and effort in carefully reviewing our paper and for raising your score. As you suggested, we will state explicitly in §4.4 that routing gains do not come mainly from the ordering constraint. We sincerely appreciate your thoughtful and constructive feedback, which has helped us strengthen the work.

Thank you,

Authors

Reviewer_yo5tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether predictive uncertainty decreases as models receive progressively richer inputs. It shows that decreasing average entropy can conceal frequent violations at the sample level, and proposes MUR-U and MUR-AU to encourage monotonic uncertainty decrease. Experiments on natural images and chest X-rays demonstrate substantial reductions in these violations, with routing benefits that vary across methods and datasets.

Critical strengths

The paper raises a useful question about how uncertainty should change as more information becomes available, and proposes simple, direct ways to address it. The routing results show that the proposed methods can improve the trade-off between acquisition cost and predictive performance in several settings. This gives the work practical value for deciding when the available information is sufficient and further acquisition can stop.

Critical weaknesses
  1. The calibration trade-off is more substantial than the paper’s discussion suggests. The concern was rasied from Figure 8 in Appendix B.2 which shows calibration degradation at the lowest information level. To further check whether the richest-input predictions are also affected, I examined the result files included with the supplementary code. The file uncertainty_quality_by_level.csv shows that MUR-U worsens highest-information ECE across all four evaluated settings; on ImageNet, ECE increases from 0.0481 to 0.0962. Interestingly, MUR-AU encourages monotonicity indirectly through softened distillation and better preserves calibration at the highest information level, suggesting that the more direct enforcement of entropy ordering in MUR-U may contribute to the calibration degradation. My concern is that this direct penalty may alter predictive uncertainty simply to satisfy the desired ordering, without making it more faithful to prediction errors. It may artificially inflate uncertainty at lower information levels or encourage overconfidence at higher levels. The calibration results suggest such distortions, but the paper does not sufficiently explain how they arise or why the resulting uncertainty estimates should be considered more useful.
  2. The source of the routing gains remains unclear. Could the authors clarify how much comes from improved uncertainty monotonicity versus better low-information predictions through distillation? This distinction matters for assessing the contribution of the proposed regularization.
Questions for authors
  1. See weakness.
2 comments
AuthorsOpenReview ↗

Response to Reviewer yo5t (1/2)

Dear Reviewer yo5t,

Thank you for the careful and constructive review, and especially for examining our released result files. We are encouraged that you found the question useful, the methods simple and direct, and the routing results practically valuable. Your analysis helped us make the paper stronger: it led us to correct the reported MUR-U runs, test whether MUR's uncertainty remains faithful to errors, and add baselines that separate the effect of ordering from that of distillation. All results are means over five seeds. We address each point below.

[W1] MUR-U worsens richest-level calibration (e.g., ImageNet: 0.048 → 0.096), and the entropy penalty may distort uncertainty, e.g., by inflating it at lower levels or encouraging overconfidence at higher levels, rather than making it more faithful to prediction errors.

Thank you; your observation was correct.

Richest-level calibration. Your analysis led us to find that we had reported MUR-U results from the wrong runs, in which the richest-level stop-gradient of Eq. (9) was disabled (General Response, [3]). We sincerely apologize. With the corrected configuration, the takeaways remain the same: richest-level ECE is now close to CE/BCE in all four settings (ImageNet-1k: from 0.096 to 0.057, vs. 0.048 for CE), and violations decrease further (ImageNet-1k: from 16.8% to 6.5%).

Faithfulness to prediction errors. To test whether MUR merely changes entropy to satisfy the ordering, we measured whether its confidence still separates incorrect from correct predictions:

Error-detection AUROC, max-probability score used for routing (poorest / richest level)ImagenetteImageNet-1kCXR-3CXR-14
CE/BCE.898 / .991.822 / .874.722 / .745.813 / .824
MUR-U (corrected).887 / .988.822 / .873.722 / .753.817 / .833
MUR-AU.883 / .986.816 / .872.720 / .751.808 / .832

AUROC stays within 1.5 points of CE/BCE at the poorest and richest levels (up to 3.2 points lower at intermediate natural-image levels). The confidence used for routing therefore remains informative about errors. MUR's uncertainty also behaves like a radiologist's in our blinded clinical study (General Response, [2]).

Calibration at poorer levels. Our primary goal is a reliable uncertainty signal for acquisition decisions (General Response, [1]). Where calibrated probabilities are needed downstream, post-hoc temperature scaling is complementary: it reduces Imagenette MUR-AU's poorest-level ECE from 0.450 to 0.063, and brings corrected MUR-U to 0.036 vs. 0.056 for CE/BCE on ImageNet-1k (a gap remains for MUR-AU on ImageNet-1k). However, per-level scaling does not preserve the ordering: scaling each level of MUR-AU independently raises Imagenette violations from 4.7% to 74.7%. We therefore recommend using MUR's original uncertainty for routing and calibrating only the selected level's probabilities after stopping.

[W2] How much of the routing gain comes from improved monotonicity versus better low-information predictions through distillation?

Thank you; this is the key question for assessing our contribution (see also General Response, [4]). We added two baselines that improve poorer-level predictions without ordering uncertainty: standard distillation (MUR-AU with α = 1) and continued CE/BCE training for the same number of epochs.

Violations (%) / routing HV gain (%)ImagenetteImageNet-1kCXR-3CXR-14
CE/BCE90.4 / 073.5 / 049.8 / 054.2 / 0
Continued CE/BCE92.1 / −3.178.4 / +17.560.3 / +0.959.4 / +1.8
Distillation (α = 1)77.8 / −1.074.4 / +18.948.6 / +1.758.5 / +2.1
MUR-U (corrected)0.7 / +2.16.5 / +13.00.0 / −1.10.0 / +0.3
MUR-AU4.7 / +6.912.3 / +18.90.0 / +1.30.0 / +1.7
  • Neither baseline yields reliable uncertainty ordering: violations remain at 49–92% across settings, compared with 0–12.3% for MUR-AU. Thus, extra training or standard distillation does not recover the ordering that MUR explicitly encourages.
  • MUR-AU achieves this without sacrificing routing utility: on Imagenette, at similar poorest-level accuracy (86.2% vs. 86.7%), MUR-AU improves routing (+6.9% vs. −1.0%) and reduces violations (4.7% vs. 77.8%) relative to distillation. On ImageNet-1k, it matches distillation's +18.9% routing gain with far fewer violations (12.3% vs. 74.4%). CXR routing differences are within one standard deviation.
  • The benefit is not solely due to distillation: MUR-U, which has no distillation term, also reduces violations and improves routing on both natural-image datasets.

Better accuracy and routing therefore do not by themselves yield a reliable uncertainty ordering; MUR provides it while maintaining competitive routing.

AuthorsOpenReview ↗

Response to Reviewer yo5t (2/2)

Additional analysis: does the ordering matter for stopping decisions? ([W2])

Routing thresholds the max-probability confidence, whereas violations are measured with entropy. We therefore checked whether MUR also orders the routing confidence, and how much acquisition cost is spent after this confidence has dropped (averaged over all thresholds):

%Imagenette: violationsImagenette: cost after a dropImageNet-1k: violationsImageNet-1k: cost after a drop
CE90.567.878.962.8
Continued CE92.070.782.569.4
Distillation (α = 1)78.266.579.268.7
MUR-U (corrected)1.93.933.637.5
MUR-AU7.716.127.029.5

Violations here use the routing confidence.

  • MUR orders the confidence that routing actually uses, not only the entropy it regularizes.
  • Under CE/BCE and both baselines, about two thirds of acquisition cost is spent after the confidence has dropped, i.e., after an acquisition left the model less certain. MUR reduces this to 4–38%.
  • On ImageNet-1k, distillation matches MUR-AU's routing gain, but its stopping decisions remain driven by these reversals, whereas MUR makes the stopping signal consistent across levels.

Thank you again for the careful analysis. We will incorporate all changes into the revised PDF, which we will upload by 15 November. Your feedback has made the paper stronger, and we believe the corrected results and new analyses address your concerns. We would be glad to discuss further, and we hope you will consider raising your score.

Thank you,

Authors

Reviewer_43ZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces the problem of aggregate monotonicty illusion, where images from different information levels (degraded versions of the same image) usually have more predictive uncertainty on aggregate but always per instance. They motivate implications for cost based decisions on information acquisition for higher quality image, which is typically based on instance level. They propose two variants of auxiliary loss terms to combat this issue: the first being a hinge loss to preserve margin between information levels and the second being a knowledge distillation term that enforces class ordering between levels. They evaluate three datasets (two versions of imagenet and Chest x-ray) and find the proposed loss decrease montonicity violations compare to standard cross entropy with temperature scaling and one baseline with acceptable tradeoff in classification accuracy.

Critical strengths
  • The paper writing is clear and the problem setting is easy to follow
  • The method is simple and intuitive and seems to be effective for the state problem
  • The empirical results are convincing on the specific datasets evaluated although specific to imagenet and CXR
Critical weaknesses
  • The problem seems related to superresolution/MR acquisition and the monotonicty violation could be artifact of the noise corruption or feature extraction. Missing comparison to feature pyramids which are trained on multiple information levels to see still entropy violations
  • lack of formal convergence guarantees for ordering, informativeness levels is vaguely defined and only certain type of image corruptions are measured. Only natural images and chest x-rays are evaluated
  • There is lack of deeper understanding of the cause of the entropy violations and if this is general problem or artifact of the training distribution or CNN architecture. There is likely confound between easy and difficult classes/instances on predictive uncertainty.
  • More baselines could be evaluated (only one external method is compared)
Questions for authors
  • what causes aggregate monotonicty illusion? how much does the violation depend on image characteristics? how to account for bias in learned feature encoder? e.g. adversarial images? How to control for pooling artifacts and training resolution biases?
  • how does coherence depend on number of classes? what about extreme classification settings? Does it reduce aggregate monotonicity?
  • Is there connection to feature pyramids? Do FPN still suffer from monotonicity illusion?
  • How is the threshold chosen? Why not calibrate the threshold at each information level for each class and dataset?
  • How does MUR-AU compare to feature pyramid network architecture? Why use one over the other? What about other superresolution techniques such as GAN or Diffusion models?
  • what is the closest works to montoncity problem? Which other works study the information montonicity problem? Why are they not compared?
  • Why do models learn different class orderings at different information levels? What are the examples/classes with highest monotonicity violation? More general insight and analysis should be provided.
  • how does coherence depend on number of classes? what about extreme classification settings? Does it reduce aggregate monotonicity?
  • how to generalize beyond images and classification tasks?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 43Zp (1/4)

Dear Reviewer 43Zp,

Thank you for the thoughtful and detailed review. We are encouraged that you found the writing clear and the problem setting easy to follow, the method simple and intuitive, and the empirical results convincing. Your questions helped us strengthen the paper: we conducted a blinded clinician study testing whether violations come from the images or the model, analyzed violations by difficulty, confidence and class, and added two new baselines. All results are means over five seeds. We address each point below.

[W1] The problem seems related to super-resolution/MR acquisition, and the monotonicity violation could be an artifact of the noise corruption or feature extraction. Missing comparison to feature pyramids, which are trained on multiple information levels, to see whether entropy violations persist.

Thank you for raising this important alternative explanation. We examined it with four new analyses, which together indicate that violations arise mainly from how models are trained rather than from the degraded inputs or the feature extractor.

  1. A radiologist rarely shows them, and MUR behaves like the radiologist. In our blinded clinician study (General Response, [2]), the radiologist became more uncertain on the clean image in only 7% of studies and MUR in 0–3%, compared with 50% for BCE (fixed by case selection).

  2. They are not caused by added noise, and they occur even when the model is right at both levels. The natural-image levels differ only in resolution; no noise is added. Moreover, violations persist even when the model predicts the correct class on both the poorer and the richer image, so the phenomenon is not restricted to transitions involving prediction errors:

    Uncertainty rises although both predictions are correct (%)ImagenetteImageNet-1kCXR-3CXR-14
    CE/BCE37.928.747.650.7
    MUR-AU0.72.71.01.8

    CXR per finding.

  3. They are not tied to one feature extractor. They appear with both ResNet-50 and DenseNet-121, and each level has its own predictor trained at that level's resolution, without parameter sharing (App. A.2).

  4. Changing only the training objective removes most of them. With the same inputs and architecture, MUR reduces violations substantially, whereas extra training and plain distillation do not ([W4]).

Feature pyramids, super-resolution and MR acquisition. Thank you for these connections; MUR is complementary to all three. An FPN fuses multiscale features of a single input, whereas in our setting the richer observation does not yet exist when the system decides whether to acquire it, so an FPN could serve as each level's backbone, with MUR on top. Super-resolution and generative models enhance the poorer input by synthesizing plausible detail, whereas MUR decides when the real, richer observation is worth acquiring; since synthesized detail cannot replace information that was never measured, the two are complementary. In MR acquisition, learned policies optimize which measurements to acquire, while MUR regularizes the uncertainty signal used to decide when to stop, so the two are complementary. We will add this discussion to Related Work.

AuthorsOpenReview ↗

Response to Reviewer 43Zp (2/4)

[W2] Lack of formal convergence guarantees for ordering; informativeness levels are vaguely defined; only certain types of image corruptions are measured; only natural images and chest X-rays are evaluated.

Thank you for these important points.

Guarantees. Ordering is not the only objective: a useful predictor must balance reliable uncertainty ordering with high accuracy at every level, and the right balance depends on the task. For example, in critical clinical tasks it is acceptable to acquire more information, as long as the system stops only when it is confident and correct, so a reliable ordering matters most. In settings where acquisition cost is the main bottleneck, high accuracy from cheaper observations may matter more than strict ordering. Enforcing ordering as a hard constraint with formal guarantees could therefore sacrifice accuracy where it matters, and richer evidence can also legitimately increase uncertainty for some samples. We therefore designed MUR as a soft regularizer, whose strength (λ for MUR-U, α for MUR-AU) sets this balance for the task at hand. Empirically, MUR reduces violations in all four settings while maintaining accuracy and routing. We will state this design choice explicitly in Sec. 3.

Informativeness. We define informativeness operationally through the acquisition process (Sec. 2.1; App. A.1): each level is a richer observation of the same sample, i.e., more useful for the target task, and accuracy increases across levels in every setting (Fig. 2). The clinician study independently confirms this for CXR (50% vs. 22% of present findings detected).

Corruptions and domains. We deliberately studied realistic information hierarchies rather than arbitrary corruptions: resolution for natural images, and underpenetration, a clinically motivated acquisition difference, for chest X-rays. Our evaluation spans multiclass and multilabel tasks, 3 to 1000 classes, and two architectures. MUR applies to any ordered acquisition process ([Q9]). We plan to evaluate it on real clinical data at a large hospital and will add this scope discussion to the paper.

[W3] Lack of deeper understanding of the cause of the entropy violations, and whether this is a general problem or an artifact of the training distribution or CNN architecture. There is likely a confound between easy and difficult classes/instances on predictive uncertainty.

Thank you for this insightful suggestion; we added analyses by difficulty, confidence and class.

Why violations arise. Each level's predictor is trained with CE on its own inputs. CE encourages calibration on average, but nothing links the predictions for the same sample across levels. Average entropy therefore decreases because accuracy increases, while per-sample entropies move independently. Consistent with this, more CE training does not help (Imagenette: 90.4% → 92.1% violations), whereas linking levels with MUR largely removes them.

Not driven by difficult instances. We group samples by whether the prediction is correct at all, some or none of the levels. Under CE/BCE, violations (%) are frequent even for easy samples (for hard samples, some increase in uncertainty is expected):

DatasetMethodEasy (correct at all levels)Medium (correct at some)Hard (correct at none)
ImagenetteCE91.982.4100.0
MUR-AU2.218.553.4
ImageNet-1kCE79.968.080.8
MUR-AU15.210.113.0
CXR-3BCE47.647.751.8
MUR-AU1.00.52.6
CXR-14BCE50.742.955.4
MUR-AU1.81.24.6

CXR per finding.

By confidence at the poorest level, CE violations (%) increase rather than decrease:

DatasetMethodQ1 (least)Q2Q3Q4Q5 (most)
ImagenetteCE79.787.390.295.199.6
MUR-AU8.85.73.12.33.6
MUR-U2.40.70.30.10.1
ImageNet-1kCE61.467.172.478.687.8
MUR-AU4.35.38.013.230.9
MUR-U6.36.56.66.26.9

MUR-U is the corrected version.

Violations are therefore not concentrated among uncertain or difficult samples. MUR substantially reduces them in every quintile, although MUR-AU's residual violations rise for the most confident ImageNet-1k samples, whose very low initial entropy leaves little room to decrease.

Not class-specific. Under CE/BCE, violations are high for every class:

Per-class violations (%), CE/BCEImagenette (all 10)ImageNet-1k (top 10)CXR-3 (all 3)CXR-14 (all 14)
Range85–9592–9543–5443–60

Together with the two architectures, two domains and the clinician study, these results support the explanation that independently trained per-level predictors drive the violations. We will add these analyses to the appendix.

AuthorsOpenReview ↗

Response to Reviewer 43Zp (3/4)

[W4] More baselines could be evaluated (only one external method is compared).

Thank you for this suggestion. In addition to Gatekeeper and temperature scaling (per-level and joint), we added two baselines that improve poorer-level predictions without ordering uncertainty: plain cross-level distillation (MUR-AU with α = 1) and continued CE/BCE training for the same number of epochs.

Violation rate (%) / HV gain over CE/BCE (%)ImagenetteImageNet-1kCXR-3CXR-14
CE/BCE90.4 / 073.5 / 049.8 / 054.2 / 0
Continued CE/BCE92.1 / −3.178.4 / +17.560.3 / +0.959.4 / +1.8
Distillation (α = 1)77.8 / −1.074.4 / +18.948.6 / +1.758.5 / +2.1
MUR-AU4.7 / +6.912.3 / +18.90.0 / +1.30.0 / +1.7

Both baselines improve routing on ImageNet-1k, but neither fixes the ordering: violations remain at 49–92% in every setting. MUR-AU reduces violations while matching or improving routing relative to CE/BCE in all four settings. We discuss the closest prior work in [Q6].

We tried to include all baselines we could identify as closest to our setting. If the reviewer has specific methods in mind that we may have missed, we would be very grateful for pointers and would be glad to include them.

[Q1] What causes aggregate monotonicity illusion? How much does the violation depend on image characteristics? How to account for bias in the learned feature encoder, e.g., adversarial images? How to control for pooling artifacts and training resolution biases?

Cause and image characteristics: please see [W3]. Feature encoder, pooling and resolution: please see [W1]. Adversarial images are a different robustness setting, as our levels are natural acquisitions of the same sample; we will state this scope and see it as an interesting direction.

[Q2] How does coherence depend on the number of classes? What about extreme classification settings? Does it reduce aggregate monotonicity?

Violations are substantial at every scale we study, with no monotonic trend in the number of classes, and MUR reduces them in every setting:

SettingClassesCE/BCE violations (%)MUR-AU violations (%)
CXR-3349.80.0
Imagenette1090.44.7
CXR-141454.20.0
ImageNet-1k100073.512.3

Extreme classification is a natural extension that we leave to future work. Regarding the last question, MUR does not change the aggregate trend, which is preserved since accuracy is maintained. Instead, it reduces per-sample violations, so that individual samples follow the aggregate trend, which removes the illusion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank the reviewers for their thoughtful and constructive feedback, which has helped us strengthen the paper. We are encouraged that they found the problem clear and useful, the method simple and intuitive (43Zp, yo5t), and the evaluation broad (HRGa). Before our individual responses, we would like to highlight four important points for all reviewers.

[1] What is the goal of MUR?

When information is acquired step by step, as when a clinician orders a costlier test only if a simpler one is inconclusive, the decision to acquire more rests on current uncertainty. For uncertainty to guide this decision reliably, it should decrease as information is added.

  • Standard training does not ensure this. Uncertainty decreases on average, yet increases for 49.8–90.4% of individual samples, so it cannot reliably guide when to stop.
  • MUR makes uncertainty a reliable signal, reducing these violations to 0–12.3% (MUR-AU).
  • Reliability does not come at the expense of utility: MUR-AU's routing is on par with or better than CE/BCE in all four settings, and its error detection stays close to CE/BCE.
  • MUR is broadly applicable. It requires only predictive distributions across information levels, so it applies wherever information is obtained incrementally under an acquisition policy. Future extensions include clinical data at a large hospital and RL environments where an agent selects its next observation.

[2] Are violations an artifact of the images, or could richer inputs legitimately increase uncertainty? (43Zp, HRGa)

We tested this with a blinded clinical study. A radiologist rated 30 CXR-3 studies from 1 (definitely absent) to 5 (definitely present). Each was shown as an underpenetrated and a clean image, as the models see them, in random order and blinded to labels, model outputs and pairings. A violation is lower confidence (distance from 3) on the clean image. Per finding, we randomly selected five BCE-violating and five non-violating cases.

Violation rate, % (95% CI)MassConsolidationPneumothoraxOverall
Radiologist10 [2–40]0 [0–28]10 [2–40]7 [2–21]
BCE50 [24–76]50 [24–76]50 [24–76]50 [33–67]
MUR-AU10 [2–40]0 [0–28]0 [0–28]3 [1–17]
MUR-U (corrected)0 [0–28]0 [0–28]0 [0–28]0 [0–11]

BCE's 50% follows from the case selection, not a population estimate.

The clean images were more informative: the radiologist identified 50% (9/18) of present findings on them versus 22% (4/18) on degraded ones. Yet the radiologist rarely lost confidence on them (2/30), so at least 13 of the 15 cases where BCE violates are not reflected in their judgments. MUR behaves like the radiologist (1/30 and 0/30). This suggests that frequent BCE violations come from the model rather than the images. This is a one-reader pilot; monotonicity remains a soft inductive bias.

[3] MUR-U worsens richest-level calibration. Does the penalty distort uncertainty? (yo5t)

We are grateful to Reviewer yo5t for carefully examining our result files. This led us to find that we had reported MUR-U results from the wrong runs, in which the richest-level stop-gradient of Eq. (9) was disabled. This let the penalty sharpen the richest-level predictor, contributing to the calibration drop the reviewer observed. We sincerely apologize for this error. We reran MUR-U as described, with the original tuning procedure:

DatasetRichest ECE, CE/BCEMUR-U richest ECEMUR-U sample-level violations (%)MUR-U routing gain (%)
Imagenette0.0040.014 → 0.0030.8 → 0.7−5.9 → +2.1
ImageNet-1k0.0480.096 → 0.05716.8 → 6.5+12.4 → +13.0
CXR-30.0270.049 → 0.0280 → 0+1.1 → −1.1
CXR-140.0140.038 → 0.0130 → 0+1.0 → +0.3

MUR-U: reported → corrected. The takeaways are unchanged, with richest-level calibration close to CE/BCE in every setting, fewer violations on ImageNet-1k, and better routing on both natural-image datasets; CXR routing changes are within one standard deviation. MUR-AU and all baselines are unaffected. Error-detection AUROC also stays close to CE/BCE (see our reply to yo5t).

[4] Does MUR's benefit come from distillation or extra training? (yo5t, HRGa)

We agree this distinction is important and added standard distillation (α = 1) and continued CE/BCE training. Both leave violations at 49–92% in every setting. On ImageNet-1k, distillation matches MUR-AU’s +18.9% routing gain, but with 74.4% vs. 12.3% violations. Neither yields a reliable uncertainty signal, which MUR provides.

We also thank Reviewer HRGa for carefully checking our numbers, which helped us clarify our definitions and make several statements more precise. We will address all points in the revised PDF, with changes in blue, and upload it by 15 November.

Thank you,

Authors

AuthorsOpenReview ↗

Follow-up to General Response: Revised Manuscript with Additional Results

Dear Reviewers,

Thank you for the careful and constructive reviews. We are encouraged that you found the problem highly relevant, the approach simple and intuitive, and the evaluation broad. In response to your feedback, we further strengthened the paper along the dimensions you highlighted: we conducted a blinded radiologist study, expanded our analysis of uncertainty violations and their repair, evaluated additional baselines and uncertainty reliability, and clarified the methodology and broader applicability of MUR.

Following our earlier general and individual responses, we have uploaded the revised manuscript incorporating these additional results and clarifications. All revisions are highlighted in blue for ease of reference, and quantitative results are reported over five seeds. We summarize the key updates and their locations below.

  1. Blinded radiologist study (43Zp, HRGa): We conducted a blinded one-reader pilot on 30 paired chest X-rays to examine whether richer observations legitimately increase uncertainty. The radiologist rarely exhibited violations (7%), consistent with MUR (0–3%), providing complementary clinical evidence that frequent baseline violations are not inherent to the observations (Section 4.2, Table 1; Appendix E).

  2. Understanding and repairing violations (43Zp, HRGa): We analyzed violations across prediction correctness, sample difficulty, initial confidence, and classes, including qualitative examples. These analyses show that violations persist even when predictions are correct at both levels, while MUR repairs violations without sacrificing correctness and rarely introduces new violations (Section 4.3, Figure 3; Appendices B.3–B.4).

  3. Corrected MUR-U results and uncertainty reliability (yo5t): We corrected the reported MUR-U experiments using the intended richest-level stop-gradient. The corrected results maintain richest-level calibration close to CE/BCE, while additional error-detection analyses show that confidence remains informative about prediction errors (Sections 3.1 and 4.3; Appendix B.2).

  4. Additional baselines and ablations (yo5t, HRGa): We evaluated continued CE/BCE training and standard distillation, demonstrating that improvements in predictive performance and routing do not automatically produce coherent uncertainty. We also showed that MUR improves the coherence of the confidence directly used for acquisition decisions (Section 4.4; Appendix B.6).

  5. Definitions and methodological clarifications (43Zp, yo5t, HRGa): We clarified the distinctions among sample-level, transition-level, and per-finding violation rates and refined the corresponding analyses and claims. We also clarified the MUR-AU formulation, training protocol, computational requirements, and routing-cost definitions, and expanded the discussion of MUR's applicability to policy-driven information acquisition (Sections 2–6; Appendices A, C, D).

Thank you again for your time and effort in reviewing our paper. We sincerely appreciate your thoughtful feedback and believe the additional experiments and analyses substantially strengthen the work. We hope you find our responses and the revised manuscript satisfactory and will consider raising your scores.

Thank you,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.