acceptodds
Under review as a conference paper at ICLR 2027

Capability Is Not Utility: Context-Dependent Routing of Frozen Black-Box MIL Experts

Abstract

Routing among frozen heterogeneous experts is challenging because expert capability does not directly determine intervention utility. In computational pathology, independently trained multiple-instance learning (MIL) models often exhibit complementary strengths across patients, tasks, and expert sets, yet their relative usefulness can change with the evaluation context. We identify two key mismatches in capability-based routing: capability estimated from historical observations may not translate into anchor-relative decision utility, and expert preferences learned in one environment may not remain stable after transfer. We introduce EpiAtlas-Safe, a post-hoc routing framework for frozen black-box MIL experts. EpiAtlas-Safe combines episodic expert-set training, a two-region Capability Atlas for modeling heterogeneous expert behavior, and selective class-balanced safe coupling (SCSC) to align routing decisions with anchor-relative utility while reducing unreliable routing decisions. The framework operates only on class-probability predictions, enabling plug-compatible coordination of independently trained experts without accessing internal representations or joint retraining. On patient-disjoint CAMELYON+ and BRACS evaluations, EpiAtlas-Safe improves patient-weighted slide-level Macro-F1 over a matched capability-transfer router by 1.30 and 2.28 percentage points, respectively, with positive dataset-level gains across all fixed router seeds. In a separately held-out BRCA evaluation, it achieves a 1.08-point improvement on ER while remaining near parity on the prespecified ER/PAM50 primary-task average. Ablation studies support the contributions of SCSC and the Capability Atlas. Additional analyses of expert-ranking changes, utility variation, and residual oracle headroom support modeling expert capability as a context-dependent decision quantity rather than a static global score. Together, these results support viewing frozen-expert routing as context-dependent Anchor-relative intervention selection rather than direct transfer of a static capability ranking. Code is available at https://anonymous.4open.science/r/epiatlas-safe-review-6D07/README.md.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Transfer Learning, Model Routing, Learning to Defer, Computational Pathology, Multiple Instance Learning, Frozen Black-Box Experts

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DNnJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines post-hoc probability-level combination of individually learned, fixed MIL experts compared to a chosen Anchor. EpiAtlas-Safe is built using expert set training using episodes, a two-region capability atlas, and selective, class balanced safe coupling, which punishes errors. Assessment is done by using patient-disjoint CAMELYON+ and BRACS split datasets, patient held-out TCGA-BRCA task evaluation, and an exploratory large expert group. The paper also explores variations in expert ordering and label dependent oracle headroom in order to prove that historic expert capability cannot directly dictate intervention utility after transfer.

Critical strengths
  1. The intervention-based formulation helps with model reuse. The difference between expert correctness and the improvement relative to some baseline prediction makes the cost of the dangerous overrides clear. The black box probability API can be used to synchronize whole MIL models without joint training or internal state, which is a relevant practical scenario.

  2. Negative results are reported transparently in the paper with appropriate evaluation measures. The use of patient normalization, patient clustering bootstrap confidence intervals, fixed held out BRCA evaluation, and well-labeled non-deployable oracle analysis is helpful. It is revealed that static Reliability weighting wins on BRACS, the BRCA average is almost the same, and pool expansion degrades the router.

Critical weaknesses
  1. Primary performance fails to convincingly outperform simpler deployable approaches. The CAM Macro-F1 is 0.6073 compared to 0.6047 for equal fusion, a fairly marginal improvement. On the BRACS data set, the proposed approach obtains worse performance than the Anchor and the Reliability weighting approaches. This does not indicate consistent superiority relative to simpler fusion or using the Anchor.

  2. Uncertainty assessment and supplementary analyses do not support transfer. For gains against Capability-Transfer, the bootstrap confidence intervals cross zero on both primary data sets. For ER/PAM50 Macro-F1, the prespecified mean is 0.7103 compared to 0.7114. Similarly, PR/HER2 scores are inferior. Expanding pools in exploratory analysis degrades performance, restricting evidence of generalization.

  3. Component evidence does not warrant the extra complexity of the router. The approach involves capability descriptors, local clustering, episodic training, ranking supervision, and a risk penalty. But the atlas provides a tiny improvement in CAM Macro-F1, and episodic training has a negative impact in BRACS ablations. Possibly a simpler utility correction/breakage model could serve as the control case for validating such complexity on grounds of competence versus utility distinction.

  4. The safety surrogate is not associated with the prediction breakage properly when applied in deployment. Eq. 9 punishes the weighted sum of mass of incorrect expert predictions given correct Anchor. However, deployment makes use of argmax based on fused probability. There is no demonstration that controlling the surrogate ensures enforcing the reference budget on the real fused prediction breakage. Neither held-out budget violation estimates nor justification of the relationship between these variables can be provided.

Questions for authors
  1. Is there any preset setting for which it shows a guaranteed gain over equal fusion, Reliability weighting, and validation-selected expert or logistic stacker under same information and split settings?

  2. What relation does Equation 9 have with the breakage of the argmax in the fused output through the weighted expert error approximation?

  3. In light of the BRACS and expansion of pool degradations, what justifies the episodic training and the atlas over the simpler utility-based correction/breakage model, using the same probability estimates?

Reviewer_wo2DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EpiAtlas-Safe, a post-hoc routing framework for coordinating frozen heterogeneous MIL experts in computational pathology. The central motivation is that expert capability does not necessarily translate into anchor-relative intervention utility, particularly across different patients, tasks, and expert compositions. To address this issue, the authors introduce episodic expert-set training, a two-region Capability Atlas for context-dependent capability estimation, and Selective Class-Balanced Safe Coupling for encouraging beneficial interventions while penalizing harmful ones. The framework operates on expert probability outputs without requiring access to internal model representations. Experiments on CAMELYON+, BRACS, and TCGA-BRCA investigate routing performance, expert utility shifts, and residual intervention opportunities.

Critical strengths
  1. The paper addresses an important problem in heterogeneous expert coordination by distinguishing expert competence from anchor-relative intervention utility.

  2. The proposed framework operates on frozen experts through their probability outputs without requiring architectural modifications or joint retraining.

  3. The paper evaluates multiple datasets and expert compositions, provides component-level ablations, and investigates expert-ranking changes, correction/breakage behavior, and Oracle intervention headroom.

Critical weaknesses
  1. My main concern lies in the insufficient differentiation from existing expert-routing approaches. The main components rely on relatively established techniques, including expert-subset sampling, clustering-based local statistics, similarity-based capability estimation, pairwise ranking losses, and weighted probability fusion. Closely related research on learning to defer, selective prediction, and routing to unseen experts has already investigated context-dependent expert selection and intervention decisions. The paper should more clearly establish the distinctive methodological contributions of EpiAtlas-Safe beyond adapting existing routing principles to black-box MIL setting.

  2. The main experimental results do not convincingly establish the superiority of EpiAtlas-Safe over simpler expert combination strategies. On CAM, EpiAtlas-Safe achieves a Macro-F1 of 0.6073, only marginally exceeding Equal Fusion (0.6047). More importantly, on BRACS, it underperforms both the Anchor (0.4251) and Reliability weighting (0.4457), achieving only 0.4143. Although the proposed method improves over Capability-Transfer, this comparison alone is insufficient to demonstrate the practical advantage of the proposed routing framework.

  3. The routing-weighted breakage risk in Eq. (9) measures the weight assigned to incorrect experts when the Anchor is correct, rather than the actual breakage rate of the final probability-fused prediction. Since the final decision is obtained by aggregating expert probabilities, these quantities are not necessarily equivalent. Moreover, the risk constraint is implemented through a soft penalty without a formal safety guarantee.

Questions for authors

See Weakness.

Reviewer_qVRDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This article investigates how to dynamically select and fuse multiple pre trained pathological classification models. The author first specifies a default model, and then determines to what extent other models should participate in the final prediction based on the characteristics of each case. The entire process does not require accessing the internal structure of these models or retraining them, but mainly relies on the class probabilities output by the models and the ability information obtained during the training phase. To this end, the author proposes EpiAtlas Safe: improving adaptability by simulating training methods with different expert combinations, describing the performance of experts in different situations using Capability Atlas, and encouraging routers to correct errors in the default model through SCSC training objectives, while reducing the situation of correcting correct results. The most noteworthy point of the paper is that a model that has performed well in the past does not necessarily mean it will be helpful in the current case or task. The author conducted experiments on CAMELYON+, BRACS, and TCGA-BRCA based tasks, and analyzed the effects of changes in expert ranking, theoretical potential improvements, and changes in the number of experts. These analyses have some value, but whether the methods themselves have truly brought about stable improvements still requires more comprehensive evidence.

Critical strengths

The author proposes EpiAtlas Safe: improving adaptability by simulating training methods with different expert combinations, describing the performance of experts in different situations using Capability Atlas, and encouraging routers to correct errors in the default model through SCSC training objectives, while reducing the situation of correcting correct results. The author conducted experiments on CAMELYON+, BRACS, and TCGA-BRCA based tasks, and analyzed the effects of changes in expert ranking, theoretical potential improvements, and changes in the number of experts.

Critical weaknesses

The difference between SCSC and regular sorting loss is not clear enough. The author emphasizes that SCSC not only distinguishes between right and wrong experts, but also considers category imbalance and additionally punishes experts who will correct the correct results of Anchors. However, from the current calculation method, when comparing experts within the same case, certain common weights and penalties may cancel each other out after normalization. In this way, the actual role played by SCSC may be very similar to the sorting loss of ordinary "putting the right expert before the wrong expert".

How much better is the method than a simple fusion strategy, and it has not been stably proven yet. On CAMELYON+, the Macro-F1 of EpiAtlas Safe is 0.6073, and the equal weight fusion is 0.6047, with a small difference; On BRACS, the method is 0.4143, which is lower than the default model's 0.4251 and the reliability weighted method's 0.4457. Although the improvement direction of both sets of data is positive relative to the matching routing baseline set by the author, the confidence intervals of the improvement both contain zero, and it is currently uncertain whether this improvement is stable enough.

  1. The cross task results indicate the existence of the problem, but do not prove that the method has solved the problem. On the TCGA-BRCA derived tasks, the average performance of the pre-set ER and PAM50 main tasks did not exceed the matching routing baseline, and the PR and HER2 tasks did not show significant advantages. After adding candidate experts, the results of CAMELYON+and BRACS showed a decrease. These phenomena strongly indicate that the usefulness of experts varies with tasks and expert combinations. But they cannot directly indicate that EpiAtlas Safe has been able to cope with this change.

The risks measured during training may not necessarily correspond to actual errors in the final prediction. The paper uses a risk indicator to measure how much weight the router assigns to erroneous experts, but the final prediction is the result of probability fusion of multiple experts. Assigning higher weights to incorrect experts may not necessarily lead to errors in the final classification; Conversely, some fusion errors cannot be explained solely by this indicator. In the appendix, the number of times CAMELYON+"corrected correct predictions" increased from 9 times in the control method to 14 times, which is also worth further discussion.

  1. The comparison with existing expert selection methods is not sufficient. The paper discusses multi expert decision delegation and routing methods when facing new experts, but the main experiments still focus on the routing baseline designed by the author and several static fusion strategies.
Questions for authors
  1. Have new risk measurement methods and generative training methods been designed, or has an effective combination of these existing technologies been proposed? At the same time, it is necessary to strengthen the comparison with the closest work to risk sensitive GFlowNet, and avoid describing methods that are larger than the actual scope of innovation

  2. Can the algorithm and implementation instructions be unified and compared under the same computational budget: selecting the optimal group each time, randomly sampling one group, taking the average of predictions after sampling multiple groups, and using ordinary deterministic feature selection

  3. Can the remaining structure and training process remain unchanged, with only the penalty for utility fluctuations turned off, and then compared with the complete model; Simple modules with similar parameter sizes and only predicting average returns can also be added as controls.

1 comment

My native language is Chinese. After reading this manuscript, I provided my opinion in Chinese and had it translated into English by AI. The editor seems to believe that this is an AI generated opinion, and they are requesting me to make further comments. But I think their viewpoint is obviously inappropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.