- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines post-hoc probability-level combination of individually learned, fixed MIL experts compared to a chosen Anchor. EpiAtlas-Safe is built using expert set training using episodes, a two-region capability atlas, and selective, class balanced safe coupling, which punishes errors. Assessment is done by using patient-disjoint CAMELYON+ and BRACS split datasets, patient held-out TCGA-BRCA task evaluation, and an exploratory large expert group. The paper also explores variations in expert ordering and label dependent oracle headroom in order to prove that historic expert capability cannot directly dictate intervention utility after transfer.
Critical strengths
-
The intervention-based formulation helps with model reuse. The difference between expert correctness and the improvement relative to some baseline prediction makes the cost of the dangerous overrides clear. The black box probability API can be used to synchronize whole MIL models without joint training or internal state, which is a relevant practical scenario.
-
Negative results are reported transparently in the paper with appropriate evaluation measures. The use of patient normalization, patient clustering bootstrap confidence intervals, fixed held out BRCA evaluation, and well-labeled non-deployable oracle analysis is helpful. It is revealed that static Reliability weighting wins on BRACS, the BRCA average is almost the same, and pool expansion degrades the router.
Critical weaknesses
-
Primary performance fails to convincingly outperform simpler deployable approaches. The CAM Macro-F1 is 0.6073 compared to 0.6047 for equal fusion, a fairly marginal improvement. On the BRACS data set, the proposed approach obtains worse performance than the Anchor and the Reliability weighting approaches. This does not indicate consistent superiority relative to simpler fusion or using the Anchor.
-
Uncertainty assessment and supplementary analyses do not support transfer. For gains against Capability-Transfer, the bootstrap confidence intervals cross zero on both primary data sets. For ER/PAM50 Macro-F1, the prespecified mean is 0.7103 compared to 0.7114. Similarly, PR/HER2 scores are inferior. Expanding pools in exploratory analysis degrades performance, restricting evidence of generalization.
-
Component evidence does not warrant the extra complexity of the router. The approach involves capability descriptors, local clustering, episodic training, ranking supervision, and a risk penalty. But the atlas provides a tiny improvement in CAM Macro-F1, and episodic training has a negative impact in BRACS ablations. Possibly a simpler utility correction/breakage model could serve as the control case for validating such complexity on grounds of competence versus utility distinction.
-
The safety surrogate is not associated with the prediction breakage properly when applied in deployment. Eq. 9 punishes the weighted sum of mass of incorrect expert predictions given correct Anchor. However, deployment makes use of argmax based on fused probability. There is no demonstration that controlling the surrogate ensures enforcing the reference budget on the real fused prediction breakage. Neither held-out budget violation estimates nor justification of the relationship between these variables can be provided.
Questions for authors
-
Is there any preset setting for which it shows a guaranteed gain over equal fusion, Reliability weighting, and validation-selected expert or logistic stacker under same information and split settings?
-
What relation does Equation 9 have with the breakage of the argmax in the fused output through the weighted expert error approximation?
-
In light of the BRACS and expansion of pool degradations, what justifies the episodic training and the atlas over the simpler utility-based correction/breakage model, using the same probability estimates?
My native language is Chinese. After reading this manuscript, I provided my opinion in Chinese and had it translated into English by AI. The editor seems to believe that this is an AI generated opinion, and they are requesting me to make further comments. But I think their viewpoint is obviously inappropriate.