acceptodds
Under review as a conference paper at ICLR 2027

Where Can Risk Be Repaired? Evidence Acquisition and Escalation for Selective Clinical LLM Decision-Making

Abstract

In clinical decision making under incomplete evidence, current methods mainly answer whether the current decision might be wrong, but they cannot tell whether to repair an error by asking for additional evidence or deferring the case to a stronger model. We propose a repairability-aware decision framework that separates current decision risk from two distinct forms of repairability. We also construct a frozen, deployable DECIDE/ASK/DEFER router. On the MIMIC-IV held-out test cohort, Always DECIDE achieves 36.39% accuracy, the deployable ASK-only policy reaches 59.02%, and the joint router reaches 62.69%, improving over ASK-only by 3.67 percentage points. In a cross-family robustness evaluation using Qwen3-8B and Qwen3-32B, we again observe the non-redundant structure of ASK and DEFER repair sets. However, the joint router does not show a statistically significant advantage over ASK-only. This suggests that the repair channel structure can generalize across model families, while the incremental gain from joint routing is model-pair dependent. We also find that increasing the ASK budget does not monotonically improve accuracy. Learned stopping reduces the mean number of questions from 5.00 to 3.18 without evidence of a decrease in accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, selective prediction, uncertainty, information acquisition, learning to defer, clinical decision making, risk calibration, adaptive inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2X1vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduce a novel way of solving the problem of ASK or DEFER which is called the repairability. And a cost-aware router about how to more effectively choose DECIDE/ASK/DEFER.

Critical strengths
  1. The idea is very novel, the paper does not stop as the three types of decisions, but also move further about the how to repair them.
  2. The experiments are quite solid.
Critical weaknesses
  1. I think in general it is a very good paper, but only the worry is that in the evidence repairability is there any theory guarantee in there?
  2. I feel the idea and way of doing this problem is really nice but only that there is no theory guarantee.
Questions for authors
  1. In the paper, the author only separate consider the problem of ASK & DEFER, and why the author did not consider to put them together?
  2. Also could authors provide more guarantee about when the repairability provides improvement.
1 comment

We appreciate the reviewer's positive assessment of the novelty and experimental design. The reviewer's main concern was the lack of a theoretical guarantee; in the revision, we (i) clarify that ASK and DEFER are jointly selected, and (ii) add a formal analysis characterizing when joint repairability-aware routing improves utility and how estimation error affects a learned value-based router.

Q1. Are ASK and DEFER considered jointly?

Yes. ASK and DEFER are estimated separately because they repair errors through different mechanisms: ASK changes the available evidence, whereas DEFER changes the decision maker. However, they are selected jointly.

In the primary frozen router, ASK and DEFER first satisfy their development-frozen eligibility thresholds; when both are eligible, the router directly compares D^Q\hat D_Q and V^H\hat V_H and selects one action (Appendix B.6). In the cost-aware formulation, this joint comparison is written explicitly as

a∗(s)=arg⁡max⁡{0, D^Q(s,M)−cQ, V^H(s,M)−cH}.a^\ast(s)=\arg\max\lbrace 0,\ \hat D_Q(s,M)-c_Q,\ \hat V_H(s,M)-c_H\rbrace.

This rule applies after the common risk gate and subject to ASK admissibility. Thus, the two repair channels are modeled separately but compete within one DECIDE/ASK/DEFER policy. We revised Section 3.4 to make this explicit.

Q2. Can repairability provide a theoretical guarantee, and when does it improve performance?

Yes, for the value-based formulation, together with a conditional bound for its learned plug-in implementation (Section 3.5; proofs in Appendix C.6). Let

gQ(s)=DQ(s,M)−cQ,gH(s)=VH(s,M)−cHg_Q(s)=D_Q(s,M)-c_Q,\qquad g_H(s)=V_H(s,M)-c_H

denote the net values of evidence repair and capability repair, with DECIDE assigned value zero, and let U(π)U(\pi) denote the expected net gain of policy π\pi over DECIDE.

Proposition 1 (Oracle dominance). The oracle joint router weakly dominates DECIDE-only, oracle ASK-only, and oracle DEFER-only because pointwise

max⁡{0,gQ,gH}≥max⁡{0,gQ},max⁡{0,gQ,gH}≥max⁡{0,gH}.\max\lbrace 0,g_Q,g_H\rbrace\ge\max\lbrace 0,g_Q\rbrace,\qquad \max\lbrace 0,g_Q,g_H\rbrace\ge\max\lbrace 0,g_H\rbrace.

Proposition 2 (When joint repairability helps). The gain over oracle ASK-only is exactly

U(πjoint∗)−U(πASK∗)=E[(gH(s)−max⁡{0,gQ(s)})+].U(\pi^\ast_{\rm joint})-U(\pi^\ast_{\rm ASK})=\mathbb{E}\left[\left(g_H(s)-\max\lbrace 0,g_Q(s)\rbrace\right)_+\right].

Hence joint routing strictly improves over ASK-only exactly when capability repair has greater net value than both evidence repair and DECIDE on a set of states with positive probability; the result relative to DEFER-only is symmetric. Joint routing therefore strictly improves over both oracle single-channel policies exactly when each repair channel has the strictly highest net value on some set of states with positive probability.

Proposition 3 (Plug-in estimation bound). For the value-based plug-in router, suppose

max⁡a∣g^a(s)−ga(s)∣≤ε(s).\max_a\left|\hat g_a(s)-g_a(s)\right|\le\varepsilon(s).

Then

U(π^)≥U(πjoint∗)−2E[ε(s)].U(\hat\pi)\ge U(\pi^\ast_{\rm joint})-2\mathbb{E}[\varepsilon(s)].

Moreover, if the oracle gap between the best and second-best actions at state ss exceeds 2ε(s)2\varepsilon(s), the plug-in router selects the oracle-optimal action at that state. The proof follows from

gπ∗≤g^π∗+ε≤g^π^+ε≤gπ^+2ε.g_{\pi^\ast}\le\hat g_{\pi^\ast}+\varepsilon\le\hat g_{\hat\pi}+\varepsilon\le g_{\hat\pi}+2\varepsilon.

These results also hold under a common risk gate: states rejected by the gate take DECIDE under all compared policies, and the analysis applies to admitted states. They characterize the value-based rule rather than the separately thresholded primary router, and Proposition 3 is conditional on the estimation error rather than a finite-sample guarantee (Section 5).

Empirical connection. In hindsight-oracle accuracy, joint routing reaches 75.84% versus 68.20% for oracle ASK-only, illustrating the complementary repair opportunities characterized by Proposition 2. In the primary deployable experiment, joint routing improves over ASK-only from 59.02% to 62.69% (+3.67 pp; 95% CI [1.53, 5.81]; exact McNemar p=0.001831p=0.001831), demonstrating a statistically significant held-out gain for the evaluated Llama model pair.

We appreciate the reviewer's recognition of the paper's strengths and hope the added theoretical analysis addresses the remaining concern about guarantees. We would be grateful if the reviewer would reconsider the score in light of the revision.

Reviewer_1DvzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a repairability-aware routing framework for clinical LLM decision making. It separately estimates whether an error can be repaired by additional evidence or by escalating to a stronger model. Models for decision risk, evidence repairability, and capability repairability are learned. The experiments on a forced four-way task with two model pairs show the improvement from joint routing.

Critical strengths
  • The idea is interesting and practically useful for applying LLMs in the medical domain. In particular, the distinction between ASK and DEFER is important.
  • The evaluation protocol is clean and careful. The patient-level splits are disjoint, and the routing components are selected on the development data and frozen before test evaluation. A second model family is also used as a robustness check.
Critical weaknesses
  • The main concern is that the clinical setting is highly restricted. A forced four-way diagnosis does not seem like a good example, and it does not represent the broader open-set diagnostic setting.
  • Regarding the evaluation, a simple confidence-based or margin-based router should be compared with the proposed method. Without such a comparison, it is difficult to determine how much additional value is provided by the repairability estimator.
  • The router uses a very limited feature set with the base-model top-1 prediction. In this setting, it is unclear whether the estimators learn a general notion of repairability or mainly exploit model-specific or class-specific characteristics.
Questions for authors
  • Do you have any additional results with more features, especially confidence, top-1/top-2 margin, and so on? Do you have any additional results beyond the forced four-way scenario?
  • Do you have any clear rationale for why the learned model can perform well even with this small set of features?
1 comment

We appreciate the reviewer's recognition of the importance of distinguishing ASK from DEFER and the careful evaluation protocol. We address the concerns about the restricted clinical setting, uncertainty-based routing, and limited estimator features.

W1. Restricted four-way diagnosis. We agree that the MIMIC evaluation is a controlled, closed-set task and does not establish generalization to open-set clinical diagnosis. We use this setting to isolate evidence acquisition and capability escalation while keeping patient-level splits, evidence availability, and model comparisons fixed. The DDXPlus experiment provides an additional diagnostic setting with sequential evidence acquisition, although it does not evaluate joint ASK/DEFER routing. We clarify these boundaries in Section 5 and do not claim clinical deployment readiness.

W2. Comparison with scalar uncertainty. We conducted an additional controlled comparison using confidence, top-1/top-2 margin, and entropy. We evaluated DECIDE/ASK (B1), DECIDE/DEFER (B2), and three-way DECIDE/ASK/DEFER (B3) policies, with thresholds selected using development data.

The strongest observed test result among the scalar-uncertainty configurations is B3 with entropy and ordering B, reaching 61.16% accuracy versus 62.69% for our frozen joint router (−1.53 pp, 95% CI [−8.9, 5.8] pp, exact McNemar p=0.46p=0.46). Although this accuracy difference is not statistically significant, the scalar policy intervenes in 298 of 327 cases (91.13%), compared with 250 cases (76.45%) for the joint router. Thus, it uses 48 additional interventions without achieving higher observed accuracy.

The development-selected B3 configuration with entropy and ordering A reaches 65.14% on development data but 59.94% on test. We report this separately from the best observed test configuration to avoid conflating development selection with retrospective test performance.

The evaluated uncertainty-only DECIDE/DEFER configurations reach 51.07% test accuracy, further illustrating the difficulty of allocating capability escalation using a single uncertainty score.

Importantly, Appendix B.3 also evaluates uncertainty as a replacement for the current-risk gate while retaining D^Q\hat D_Q and V^H\hat V_H. This entropy-gated joint router reaches 62.39%, close to the original 62.69%. Thus, uncertainty is competitive for risk detection, while the evaluated scalar-only policies provide less efficient three-way intervention allocation.

W3 / Q1. Additional estimator features. We evaluated adding likelihood-based confidence, top-1/top-2 margin, and entropy to the original age, sex, chief-complaint, and base-prediction features (Appendix B.4, Table 4).

For current risk, AUROC increases from 0.8177 to 0.8645. For evidence repairability, AUROC changes only from 0.7574 to 0.7609, with a decrease in AUPRC. For capability repairability, repair AUROC increases from 0.7892 to 0.8226 and harm AUROC from 0.7001 to 0.7294, although both corresponding AUPRC values decrease.

These results show that model confidence substantially improves risk estimation but provides mixed improvements for predicting which intervention repairs an error.

Q2. Why can a limited feature set support repairability estimation? The current prediction, chief complaint, and patient characteristics provide observable information about the model's decision and clinical context. We use these features to estimate intervention-specific outcomes rather than treating uncertainty as a direct substitute for repairability. The frozen estimators can therefore learn conditional associations with ASK and DEFER outcomes without accessing test labels or post-intervention outcomes at inference time.

However, we agree that these associations may partly reflect diagnosis-specific or model-specific behavior. The Qwen robustness evaluation reproduces the non-redundant ASK/DEFER repair-set structure, but does not establish that the learned estimators generalize beyond the evaluated models and diagnostic categories.

Summary. The additional experiments directly address the requested uncertainty and feature comparisons. They show that scalar uncertainty is competitive for detecting unreliable predictions, but the evaluated scalar-only three-way router requires more interventions for slightly lower observed accuracy. Confidence features improve risk estimation more consistently than repairability estimation. We acknowledge the restricted diagnostic setting and the remaining limitations of the learned estimators.

We appreciate the reviewer's consideration of these additions and hope they address the concerns underlying the assessment.

Reviewer_biSVOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how clinical LLMs should respond to incomplete evidence, arguing that identifying a potentially incorrect decision does not determine which intervention would improve it. It distinguishes current decision risk, evidence repairability through additional patient information, and capability repairability through escalation to a stronger model. Capability repairability explicitly accounts for both corrected errors and newly introduced errors.

The authors implement these quantities using learned estimators and construct a DECIDE/ASK/DEFER router, with thresholds, evidence-channel priorities, and action-selection rules fixed before test evaluation. The primary experiments use four abdominal diagnoses from MIMIC-IV-Ext, with temporally restricted access to vital signs, laboratory results, and imaging information.

The evaluation shows that evidence acquisition and model escalation repair overlapping but distinct sets of errors across two model families. On the Llama test cohort, joint routing improves accuracy over ASK-only by 3.67 percentage points (95% CI: [1.53, 5.81]); the corresponding Qwen improvement is smaller and statistically inconclusive. Separate DDXPlus experiments examine learned stopping, while a normalized cost analysis illustrates how intervention costs affect action selection.

Critical strengths

1. The problem is well motivated, and the method tests exactly the claim it makes.

  • The paper’s framing (risk tells you whether to intervene, not where) is a real gap in selective prediction, active acquisition, and routing work, and the three-quantity decomposition (R,DQ,VHR, D_Q, V_H) maps one-to-one onto that gap. The DECIDE/ASK/DEFER action space and the frozen routing rule (B.4) follow directly from the formulation rather than being bolted on.
  • The central premise is directly tested, not assumed: both model pairs show substantial ASK-only and DEFER-only subsets (Tables 6–7), so the two channels are complementary rather than interchangeable.
  • The harm-aware escalation term earns its place empirically. Unconditional escalation repairs 67 but harms 18 (Llama) and repairs 76 but harms 65 (Qwen) (Table 9), whereas the router’s selected DEFER subset yields 17/0 and 10/1 (Table 13). This is the most consequential finding for me: a stronger model is not a monotonically safe fallback, and selectivity pays.

2. The protocol and reporting make the scope of the findings clear.

  • Patient-disjoint splits, temporal admissibility, development-only selection, an explicit frozen rule, paired CIs with McNemar tests, and oracle results kept separate from deployable ones.
  • The authors report results that limit their own claims: the null Qwen gain, threshold sensitivity at the grid boundary (Table 5), the weak stopping estimator (AUROC 0.592, AUPRC 0.043), and that the zero-cost policy is not the primary router (F.5). This raises my confidence that the Llama result is not a tuning artifact, even if it leaves open how general it is.
Critical weaknesses

1. The main result (that jointly routing between ASK and DEFER beats ASK alone) rests on thin evidence: one small test set, one model pair, and a second model pair where the gain disappears.

  • The headline +3.67 pp improvement comes from a single held-out set of 327 patients covering only four abdominal diagnoses. Across those 327 cases, the joint router and ASK-only disagree on just 14; the joint router wins 13 of them and loses 1 (Appendix D.2). A result driven by 14 cases on one cohort is fragile, and the dataset is too small to support a general claim.
  • When the authors repeat the experiment with Qwen3-8B/32B, the gain shrinks to +0.92 pp with a 95% CI of [-0.61, 2.45], which includes zero (p=0.45p = 0.45). The authors' own explanation is telling: escalating to the stronger model yields a net +49 cases for Llama but only +11 for Qwen (Table 9). In other words, the joint router helps when the stronger model happens to be much better than the base model, which is a property of the chosen model pair rather than of the routing method. The paper shows convincingly that ASK and DEFER fix different cases; it does not show that the router can reliably take advantage of that.
  • The sequential experiment on DDXPlus does not strengthen the case. It uses a different model (Qwen2.5-14B), has no DEFER option, and uses none of the three estimators (R^\hat{R}, D^Q\hat{D}_Q, V^H\hat{V}_H) that define the method. So it only shows that a learned stopping rule asks fewer questions on that dataset. Even that result is weak: all accuracies sit between 14.75% and 17.25%, very low in absolute terms; the stopping classifier barely separates the two classes (AUPRC 0.0430.043); and the accuracy difference vs. always asking five questions has a CI of [-1, 2] pp, which cannot rule out a loss and should not be read as "no accuracy decrease."

2. The paper does not show that the learned evidence-repairability score (D^Q\hat{D}_Q) actually contributes, and the score is trained for a different action than the one the router executes.

  • A simple rule may explain most of the gain. Table 4 shows that requesting IMAGING improves correctness on average (ΔˉQ=+0.35\bar{\Delta}_Q = +0.35), while VITALS (−0.04-0.04) and LABS (-0.18) make it slightly worse. The router always requests imaging first when it is available. So the jump from 36% (never ask) to 59% (ASK-only) may come mostly from the rule "if the case looks risky, show the model the imaging report," with D^Q\hat{D}_Q adding little on top. The missing baseline is a risk-gated policy that always requests the top-priority channel without consulting D^Q\hat{D}_Q. The "R-only" row in Table 10 (50.46%) might be that baseline, but the text never says what action it takes, so the reader cannot tell. Table 5 adds to the concern: when the threshold search is widened, the best development threshold for D^Q\hat{D}_Q is τ=0\tau = 0, meaning every risky case passes the ASK check and the score effectively stops filtering.
  • D^Q\hat{D}_Q is trained on the wrong target for how it is used. D^Q\hat{D}_Q is trained to predict max⁡(0,max⁡qΔQ(q∣s,M))\max(0, \max_q \Delta_Q(q \mid s, M)), i.e., whether the best available evidence channel would fix the case (Appendix B.1). But at deployment, the router does not choose the best channel; it requests the highest-priority channel that is legal for that patient (Appendix B.4). The two coincide when imaging is available. When it is not, the router falls back to vitals or labs, which on average hurt accuracy, yet D^Q\hat{D}_Q cannot warn about this because its training label is clipped at zero and can never express "asking will make things worse." This also tilts the ASK-vs-DEFER decision: D^Q≥0\hat{D}_Q \geq 0 always, while V^H=P^(AH=+1)−P^(AH=−1)\hat{V}_H = \hat{P}(A_H = +1) - \hat{P}(A_H = -1) is signed and can be negative. Comparing them directly (ASK if D^Q≥V^H\hat{D}_Q \geq \hat{V}_H) favors ASK by construction, because only DEFER is penalized for possible harm. Training D^Q\hat{D}_Q on the net effect of the channel actually executed would fix both problems.
  • A key justifying claim is never checked. The whole framework rests on risk RR and evidence repairability DQD_Q being different quantities; the Introduction asserts their correlation is weak. But the paper never reports that correlation. This matters because R^\hat{R} and D^Q\hat{D}_Q are both logistic regressions on the same four inputs (age, sex, chief complaint, base-model prediction), and two such models can easily produce near-identical rankings. If R^\hat{R} and D^Q\hat{D}_Q are in fact highly correlated, the two-stage router collapses to a single threshold and the "where to repair" story is much weaker than presented. A single correlation coefficient on the development set would settle this.
Questions for authors
  1. Is the learned ASK score doing more than "request imaging"? Table 4 shows that IMAGING is the only channel that improves correctness on average (ΔˉQ=+0.35\bar{\Delta}_Q = +0.35), while VITALS (−0.04-0.04) and LABS (−0.18-0.18) slightly hurt, and the router always requests IMAGING first when it is admissible. This raises the possibility that most of the 36% → 59% gain comes from simply giving the model the imaging report, rather than from D^Q\hat{D}_Q.

  2. Risk vs. repairability correlation. The Introduction states that current risk and evidence repairability are weakly correlated, but no value is reported. On the development cohort, what is the correlation (Pearson or Spearman) between (a) R^\hat{R} and D^Q\hat{D}_Q, and (b) R^\hat{R} and the hindsight label DQHOD_Q^{HO}?

2 comments

We appreciate the reviewer's careful assessment and recognition of harm-aware selective escalation as the paper's most consequential finding. We address each point below using results reported in the revised manuscript.

W1. Reliability of the joint-routing gain. For Llama, joint routing improves over ASK-only by 3.67 pp (95% CI [1.53, 5.81]; 13 paired gains, 1 loss; exact McNemar p=0.001831p=0.001831). Both policies share the same risk gate and ASK mechanism, while joint routing additionally permits DEFER (23 of 327 test cases). We acknowledge that the 327-case, four-diagnosis cohort limits generalizability (Section 5).

For Qwen, the gain is smaller and not significant (+0.92 pp, 95% CI [−0.61, 2.45], p=0.453p=0.453). Proposition 2 (Section 3.5) characterizes the incremental oracle value of the capability channel relative to other actions. Unconditional escalation yields 67 repairs and 18 harms for Llama (net +49), compared with 76 repairs and 65 harms for Qwen (net +11). This difference is consistent with, but does not establish the cause of, the smaller Qwen gain.

Selected DEFER exhibits favorable observed outcomes in both model pairs: even for Qwen, where unconditional escalation provides only a limited net benefit (76 repairs, 65 harms), the selected subset yields 10 repairs and 1 harm among 13 cases, while Llama yields 17 repairs and no harms among 23 cases. These observations do not establish a general safety guarantee.

DDXPlus evaluates sequential ASK without DEFER. Learned stopping reduces mean acquisition from 5.00 to 3.18 questions, while the accuracy difference remains inconclusive (95% CI [−1.00, 2.00] pp). We do not claim established accuracy preservation.

W2 / Q1. Does D^Q\hat D_Q contribute beyond imaging priority? We could not reliably reconstruct the action rule of the original R-only baseline and therefore removed that row from Table 11 and Appendix D.1; no conclusion depends on it. The directly relevant ablation, which retains the risk gate and frozen ASK priority but removes D^Q\hat D_Q, reaches 59.63%, compared with 59.02% for learned ASK-only. Risk-gated imaging-only ASK reaches 61.16% (Appendix D.3). The current learned D^Q\hat D_Q does not clearly outperform the evidence-type heuristic, which we acknowledge as a limitation.

Adding harm-aware DEFER to imaging-only ASK raises accuracy from 61.16% to 65.44% (+4.28 pp observed). This post-hoc comparison is descriptive; we do not claim statistical significance. It shows that the observed benefit of selective escalation is not confined to the original learned ASK policy. Removing D^Q\hat D_Q from the original joint router reduces accuracy from 62.69% to 60.55% (+2.14 pp, 95% CI [−0.31, 4.59], p=0.143p=0.143), a statistically inconclusive contribution.

Target–execution mismatch. The original target uses the best admissible channel, whereas deployment executes the frozen priority channel. We evaluated a signed, action-aligned estimator that addresses both this mismatch and ASK-induced harm (Appendix C.5; repair AUROC 0.741, harm AUROC 0.790). Its router reaches 59.63%, compared with 62.69% for the original joint router (−3.06 pp, 95% CI [−6.73, 0.61]). The analysis does not establish an accuracy improvement.

Q2. Risk–repairability correlation. On the Llama development cohort, R^\hat R and hindsight repairability DQHOD_Q^{HO} have Pearson correlation 0.336 and Spearman correlation 0.296 (held-out test: 0.266 and 0.239). In contrast, the out-of-fold learned scores R^\hat R and D^Q\hat D_Q are strongly correlated (Pearson 0.797, Spearman 0.739). We now explicitly distinguish target-level non-equivalence from limitations of the current learned estimators (Section 4.2, Appendix C.4). The ASK/DEFER decision additionally uses the signed capability estimate V^H\hat V_H.

Summary. ASK and DEFER exhibit distinct repair opportunities across both model pairs. Selective capability escalation shows more favorable observed repair–harm balances than unconditional escalation, and the imaging-only comparison extends this finding to a heuristic ASK policy. The additional analyses also identify limitations of the current learned D^Q\hat D_Q estimator without undermining the observed distinction between evidence and capability repair channels.

We appreciate the reviewer's consideration of these revisions and would be grateful if they are reflected in the final assessment.

I appreciate the authors’ thoughtful responses and their transparency in reporting results that do not favor the proposed approach. Having considered the rebuttal to both my review and Reviewer 1Dvz’s comments, however, I remain unconvinced that the additional evidence addresses the central limitations. I am therefore revising my score to 2 (weak rejection).

1. The empirical support remains limited to one small cohort.

The response acknowledges the limitations of the 327-case evaluation but provides no additional cohort, diagnostic categories, or model pair to establish robustness. For Qwen, the estimated improvement remains statistically inconclusive: +0.92 percentage points, with a 95% confidence interval of [−0.61, 2.45].

2. The additional analyses do not establish the value of the learned ASK component.

The new ablations suggest that a simple imaging-priority heuristic performs comparably to, or better than, the learned router. This weakens the evidence for the paper’s central methodological contribution.

The role of the VITALS and LABS channels also remains unclear. The response does not account for their negative average effect on correctness or provide the number of cases routed to these fallback channels and the corresponding repair and harm counts. Separately, the removal of the R-only baseline because its action rule could not be reconstructed raises concerns about reproducibility.

3. Two issues highlighted by Reviewer 1Dvz further limit the conclusions.

  • Narrow diagnostic setting. Selecting among four predefined diagnoses provides limited evidence about performance in open-set diagnosis. The authors acknowledge this generalization gap, while the DDXPlus experiments do not assess joint ASK/DEFER routing. Together with the reliance on a single small cohort, this leaves the broader claims insufficiently supported.
  • Limited estimator inputs. The estimators use age, sex, chief complaint, and the base model’s top-1 prediction. As the authors acknowledge, the resulting associations may partly capture diagnosis-specific patterns or behavior particular to the evaluated models, leaving their transferability uncertain.

The problem remains worthwhile, but the current results do not convincingly demonstrate that the proposed learned repairability-aware router can capitalize on these complementary behaviors. Stronger support would require evaluation on an independent cohort or a broader diagnostic task, a breakdown of outcomes for VITALS/LABS fallback cases, and clear evidence that the learned ASK estimator improves upon the imaging-priority heuristic.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.