- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates whether predictive uncertainty can reliably guide a system in acquiring more informative observations for the same sample. It aims to improve the cost–performance trade-off by encouraging sample-level uncertainty to decrease as the information level increases. MUR-U and MUR-AU formulate training objectives based on entropy ordering and cross-level prediction distribution matching, respectively. Overall, the framework is clearly presented, and the experiments cover a reasonably broad range of settings. However, some numerical inconsistencies remain, and the current experimental design is insufficient to validate the paper's central claims.
Critical strengths
(1)The paper focuses on sample-level decisions, where average entropy may decrease even as uncertainty increases for individual samples. It uses the violation rate to measure this inconsistency and links it to the stopping policy. (2)Experiments across five random seeds evaluate the methods on Imagenette, ImageNet-1k, CXR-3, and CXR-14, covering multiclass and multilabel tasks, two network architectures, and different types of information. (3)The two methods have distinct effects. MUR-U enforces entropy ordering, while MUR-AU also transfers class distributions across information levels. On Imagenette, MUR-U achieves a violation rate of only 0.8%, yet its routing performance is 5.9% lower than that of cross-entropy training, showing that a lower violation rate does not necessarily lead to better routing. (4)Temperature scaling yields violation rates close to those of standard training, while Gatekeeper fails to maintain both low-information performance and routing gains. An additional tolerance analysis examines whether the violations result from negligible entropy fluctuations.
Critical weaknesses
(1) The abstract states that uncertainty increases for more than half of the samples when additional signals are provided. However, the reported proportion for CXR-3 is only 49.8%. In Fig. 3a, the transition-level and sample-level proportions are 26.0% and 73.5% for ImageNet-1k, and 37.1% and 90.4% for Imagenette, respectively. These two types of statistics should be clearly distinguished, and the statement in the abstract should be revised accordingly. (2) Each CXR-14 sample has only one transition between information levels. Therefore, the sum of the violation rates across groups should equal the sample-level violation rate. However, the group-level proportions in Fig. 9 sum to 51.0%, whereas Fig. 12 reports a violation rate of 54.2% under BCE. For CXR-3, the group-level proportions sum to 49.4%, which differs slightly from the reported value of 49.8%. (3) The paper claims that both MUR methods eliminate violations on CXR-3 and CXR-14. However, Appendix D reports residual violation rates of 0.1%–3.3% across CXR-3 disease categories, while Effusion and Infiltration on CXR-14 also retain violations. The authors should verify these results and avoid claims that are inconsistent with the reported evidence.
Questions for authors
(1) Equation (4) reflects a general property of averages. A decrease in average entropy does not guarantee that individual samples become less uncertain as more information is provided. Therefore, the paper's central empirical claim should be supported by the measured violation rates rather than inferred from average entropy. (2) The paper acknowledges that richer observations may increase uncertainty for some samples. However, Eqs. (9) and (10), together with the model selection rule based on the lowest validation violation rate, penalize each increase in entropy, allowing a tolerance of five percentage points only at the most informative level. Moreover, the performance gains of MUR-AU and MUR-U are only approximately 1.3% and 1.4%, respectively. The authors should further analyze the relationship between uncertainty constraints and actual routing gains, and clarify whether these constraints lead to consistent performance improvements. (3) Appendix A.2 uses 10 training epochs and 10 fine-tuning epochs for CXR, which does not support the claim that fine-tuning takes approximately half as long as training on CXR. In addition, the x-axis of Fig. 4 ends at 1 for CXR-3, while the axis range for natural images may also be inconsistent with the actual cost definition. In contrast, Fig. 13 extends the x-axis to 1.25 for CXR-14. The cost definition should be unified, and the corresponding figures and claims should be rechecked. (4) Eqs. (12) and (13) assign unit weights to the adjacent-level distribution-matching terms in MUR-AU, without including the factor used in the task loss or a penalty weight corresponding to in MUR-U. The authors should consider adding a baseline that improves predictions at lower information levels without explicitly enforcing uncertainty ordering. This would help distinguish gains from improved predictive performance from those attributable to the uncertainty monotonicity constraint.
4 comments
Response to Reviewer HRGa (2/2)
[Q2] Eqs. (9)–(10) and the selection rule penalize every entropy increase, although richer inputs can legitimately increase uncertainty for some samples. The gains of MUR-AU and MUR-U are only about 1.3% and 1.4%. Analyze the relationship between the uncertainty constraints and routing gains, and whether they lead to consistent improvements.
Thank you for this important question.
Penalizing every increase. MUR is a soft regularizer, not a hard constraint: λ (MUR-U) and α (MUR-AU) trade ordering against accuracy, so increases are discouraged rather than forbidden, and some remain (e.g., 12.3% for MUR-AU on ImageNet-1k). Two observations suggest that frequent CE/BCE increases are not only due to legitimate ambiguity: in our clinician pilot, the radiologist lost confidence on the clean image in only 2 of 30 selected cases (General Response, [2]), and the increases persist when small entropy changes are ignored (App. B.1). We will state explicitly that MUR targets systematic, not universal, monotonicity.
Size and consistency of the gains. The gains of about 1.3–1.4% are on CXR, where routing differences are within one standard deviation; on natural images they are larger:
| HV gain over CE/BCE (%) | Imagenette | ImageNet-1k | CXR-3 | CXR-14 |
|---|---|---|---|---|
| MUR-U (corrected) | +2.1 | +13.0 | −1.1 | +0.3 |
| MUR-AU | +6.9 | +18.9 | +1.3 | +1.7 |
MUR-AU therefore reduces violations in every setting without lowering routing in any.
Relationship between ordering and routing. Ordering makes the routing signal more consistent: under CE/BCE and both baselines, about two thirds of the acquisition cost on natural images is spent after the routing confidence has dropped, i.e., after an acquisition left the model less certain. MUR reduces this to 4–38% (see our reply to yo5t), so acquisition decisions rely less on a signal that reverses as information is added. Routing gains also depend on accuracy at each level; e.g., MUR-U has fewer violations than MUR-AU on Imagenette but a smaller gain, so we report both.
[Q3] Appendix A.2 uses 10 training and 10 fine-tuning epochs for CXR, which does not support the claim that fine-tuning takes about half as long. The x-axis of Fig. 4 ends at 1 for CXR-3, whereas Fig. 13 extends to 1.25 for CXR-14; the cost definition should be unified.
Thank you for catching both.
Fine-tuning time. You are right that the claim holds only for ImageNet-1k; we will correct App. A.2:
| Epochs | Imagenette | ImageNet-1k | CXR-3 | CXR-14 |
|---|---|---|---|---|
| CE/BCE training | 20 | 10 | 10 | 10 |
| MUR fine-tuning | 20 | 5 | 10 | 10 |
Cost axes. All figures use the same cost definition (App. A.3); for CXR, the exit costs are 0.25 and 1.25 in both datasets. In Fig. 4, the points at 1.25 are plotted, but the last tick label shown is 1.00. We will add this tick and unify tick labels across figures.
[Q4] Eqs. (12)–(13) assign unit weights to the distribution-matching terms in MUR-AU, without a weight corresponding to λ. Consider a baseline that improves lower-level predictions without explicitly enforcing uncertainty ordering.
Thank you for this suggestion. In MUR-U, λ weights the ordering penalty. In MUR-AU, α plays this role instead: it sets the softness of the distillation target, and thus how strongly ordering is encouraged. A separate weight is therefore unnecessary, so we fix it to 1 and tune only α; we will clarify this in Eqs. (12)–(13). At α = 1, MUR-AU reduces to plain distillation, which is exactly the baseline you suggest. We added it, together with continued CE/BCE training for the same number of epochs:
| Violations (%) / HV gain (%) | Imagenette | ImageNet-1k | CXR-3 | CXR-14 |
|---|---|---|---|---|
| CE/BCE | 90.4 / 0 | 73.5 / 0 | 49.8 / 0 | 54.2 / 0 |
| Continued CE/BCE | 92.1 / −3.1 | 78.4 / +17.5 | 60.3 / +0.9 | 59.4 / +1.8 |
| Distillation (α = 1) | 77.8 / −1.0 | 74.4 / +18.9 | 48.6 / +1.7 | 58.5 / +2.1 |
| MUR-AU | 4.7 / +6.9 | 12.3 / +18.9 | 0.0 / +1.3 | 0.0 / +1.7 |
Both baselines improve poorer-level predictions, and on ImageNet-1k also routing, but violations remain at 49–92%. On ImageNet-1k, distillation reaches the same +18.9% routing gain as MUR-AU, but with 74.4% vs. 12.3% violations. Improvements in predictive performance and routing therefore do not automatically yield ordered uncertainty, which MUR explicitly encourages.
Thank you again for your time and careful reading. We will incorporate all changes into the revised PDF, which we will upload by 15 November. Your feedback has made the paper more precise, and we believe the clarifications and additional analyses substantially strengthen the work. We hope you find our response satisfactory and will consider raising your score.
Thank you,
Authors
The authors responded better than I expected. My main concerns (W5/W7) are answered with new control experiments, so I have raised my score to 3. I still ask them to state in the paper what their own numbers show, that the routing gains do not mainly come from the ordering constraint.
Response to Reviewer HRGa: Thank you for your feedback and for raising your score
Dear Reviewer HRGa,
Thank you again for your time and effort in carefully reviewing our paper and for raising your score. As you suggested, we will state explicitly in §4.4 that routing gains do not come mainly from the ordering constraint. We sincerely appreciate your thoughtful and constructive feedback, which has helped us strengthen the work.
Thank you,
Authors
Response to Reviewer HRGa (1/2)
Dear Reviewer HRGa,
Thank you for the careful and constructive review. We are encouraged that you found the sample-level focus well motivated, the evaluation broad (five seeds, multiclass and multilabel tasks, two architectures, two information types), and the distinct effects of MUR-U and MUR-AU informative. Your detailed reading helped us make the paper more precise: we clarify measures that we had not distinguished clearly, make several statements more precise, and add baselines that separate better predictions from uncertainty ordering. All results are means over five seeds. We address each point below.
Thank you for pointing this out. The two rates measure different things, which we did not distinguish clearly: the sample-level rate counts samples whose uncertainty increases at any step, while the transition-level rate counts individual steps.
CXR has a single transition, so the two coincide.
We will define and report both throughout. Since CXR-3 (49.8%) is just below half, we will also make the abstract precise: uncertainty increases for about half or more of the samples (49.8–90.4% across settings).
Thank you for checking this carefully. The numbers are consistent, but we did not state their units: Fig. 12 counts one violation per sample (entropy averaged over findings), while Fig. 9 groups (sample, finding) pairs.
We will state the unit in both captions and explain the difference in the text.
Thank you; we agree that "eliminate" should be stated more precisely. MUR eliminates violations at the sample level (0.0%), while residuals remain for individual findings, mainly Effusion and Infiltration on CXR-14:
Highest residuals on CXR-14: Effusion (MUR-U 13.2%, MUR-AU 11.5%) and Infiltration (MUR-U 13.4%, MUR-AU 8.4%).
We will revise the claim to: "MUR eliminates sample-level violations on CXR-3 and CXR-14 and reduces per-finding violations from 43–60% to at most 13.4%", and report per-finding results in the main text.
We agree, and this is precisely the role of Eq. (4): it shows that aggregate trends cannot establish per-sample monotonicity, which is why all our claims rest on measured violation rates (Fig. 2; [W1]). We will make this explicit around Eq. (4).