acceptodds
Under review as a conference paper at ICLR 2027

MedCOBE: Diagnosing LLMs’ Collaboration Failures before Clinical Deployment

Abstract

Large language models are increasingly deployed as collaborative partners in the clinical domain, supporting clinicians through interaction. Yet their evaluation often relies on MedQA-style benchmarks that assess LLM knowledge in non-interactive, standalone settings. This evaluation-deployment gap raises a question: does an LLM's standalone performance reflect its ability to support clinicians through interaction? Across 20 LLMs, we observe large performance drops in deployment-like interactions, even on cases that the LLMs can solve in standalone settings (e.g., Claude-Opus-5 exhibits an 11.4% drop in decision accuracy). This shows that standalone evaluation alone is not sufficient to anticipate LLMs' failures in deployment. To address deployment-time failures before deployment, we introduce MedCOBE, a simulation sandbox that evaluates LLMs via deployment-like clinician–AI interactions. MedCOBE quantifies LLMs' interaction behavior: ability to properly react to clinician input in interaction, while isolating failed behavior from knowledge limitations. The MedCOBE score correlates more strongly with collaboration outcomes than standalone accuracy does (rho = 0.820 vs. 0.409). Moreover, MedCOBE produces behavioral profiles that reveal each LLM's interaction tendencies associated with failed collaboration, i.e., frequent sycophancy towards incorrect clinician hypotheses or obstruction of correct ones. We further show that these profiles can enable targeted mitigation of collaboration failure at both model and system levels, establishing MedCOBE as both an evaluation design and an actionable insight for safer deployment in clinical contexts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Clinical LLM, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VuZmOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MEDCOBE, a simulation framework for evaluating LLMs as clinical collaborators rather than as standalone decision makers. The main observation is that models that solve a case reliably on their own can still hurt the final decision once another decision maker enters the loop. MEDCOBE isolates this effect by evaluating collaboration only on cases where the model has sufficient standalone knowledge, and then characterizes interaction behavior in terms of whether the model appropriately affirms correct clinician hypotheses and challenges incorrect ones. Across 20 LLMs, the proposed score is more strongly associated with held-out collaborative outcomes than standalone accuracy, and the resulting behavioral profiles are further used to guide prompt- and critic-based interventions.

Critical strengths
  1. The paper identifies a meaningful evaluation gap. The “lost in collaboration” phenomenon is compelling: a model may answer a case correctly and consistently in isolation, yet still steer a clinician toward a worse decision during interaction. Conditioning the analysis on cases with sufficient standalone knowledge is a clean way to make this distinction.

  2. The evaluation is unusually thorough for a benchmark-style paper. The authors test 20 models, separate the behavioral evaluation set from the held-out collaboration set, study several interaction contexts, validate the LLM judges against human annotations, and include additional statistical analyses showing that the MEDCOBE score contributes signal beyond standalone accuracy.

  3. The behavioral profiles are easy to interpret. The distinction between sycophancy and obstruction is simple, but useful. Two models with similar overall collaboration scores can fail for very different reasons, and the paper makes a good case that these differences matter for mitigation.

Critical weaknesses
  1. I am not fully convinced that the Affirm/Challenge abstraction is rich enough to characterize good clinical collaboration. The framework treats affirming a correct hypothesis and challenging an incorrect one as the desired behaviors. This is reasonable at a high level, but real clinical discussion is often less binary. A correct hypothesis may still deserve qualification, additional testing, or pushback on management; an incorrect hypothesis may still contain useful partial reasoning. Treating partial disagreement as Challenge may therefore blur together clinically helpful nuance and genuine obstruction.

  2. The paper’s strongest deployment claims still rest on simulated clinicians. The simulator setup is useful for controlled experimentation, but it is still quite different from real clinical collaboration. The clinician is an LLM, the task is multiple choice, and both sides are given a fairly complete case representation. I think the results convincingly establish MEDCOBE as a pre-deployment sandbox, but they are weaker evidence that the same behavioral patterns will predict outcomes with practicing clinicians.

  3. Cross-model comparisons of SMEDCOBES_{\text{MEDCOBE}} are not entirely apples-to-apples. The score is computed only on cases where each model achieves KX=1K_X=1, which means different models are evaluated on different subsets of cases. For strong models these subsets are large, while for weaker models they can be much smaller and potentially easier or qualitatively different. The threshold analysis is useful, but it does not fully remove this case-mix issue.

  4. The human validation is encouraging, but fairly narrow. The judge validation is based on 124 responses from ophthalmology cases annotated by a single senior ophthalmologist. The agreement numbers are reasonably strong, but the benchmark spans seven specialties. I would be more comfortable with the action and validity labels if this validation covered a broader set of specialties or involved multiple clinicians.

Questions for authors

See "Critical Weaknesses".

2 comments
AuthorsOpenReview ↗

To Reviewer VuZm

Thank you for your time and effort. We appreciate your kind words about our effort to make this paper thorough. We will do our best to improve our paper based on your comments.


[R1] Response to “Affirm and Challenge” (W1):

We agree that real clinical interaction involves more nuanced behaviors. We intentionally use a simplified formulation so we can quantify collaboration failures at scale without extensive annotation. Regardless, we believe a finer-grained behavior categorization can further improve MedCOBE’s effectiveness and is a promising next step.

Regarding your concern (“A correct hypothesis may still deserve..”), our post-hoc behavior judge also assesses behaviors’ validity with reference to the clinical scenario and expert-annotated explanations, rather than solely using the correctness of the clinician hypothesis to judge the appropriateness of partner actions.


[R2] Response to “predicting outcomes with practicing clinicians” (W2):

To investigate whether MedCOBE can predict outcomes with practicing clinicians, we’d need a large-scale clinician study, which we were not able to conduct.

As an alternative, we tried to make the simulation more realistic by incorporating diverse usage contexts (Table 2). We hereby present two additional experiments that further address task realism by (i) removing the decision options (i.e., open-ended diagnosis) and (ii) not providing the AI partner with the complete case information up front:

MethodSpearman's ρPearson's r
Standalone Acc0.4090.421
MedCOBE (Open-ended)0.7160.593
MedCOBE (w/o Case Information up front)0.7000.602

While these experiments still do not guarantee human-in-the-loop outcomes, we believe they further support MedCOBE’s robustness in diverse, real-world-like scenarios.


[R3] Response to “apples-to-apples case pool” (W3):

We appreciate your keen observation. As you pointed out in Strengths, we condition the analysis on K_x = 1.0 cases to disentangle interaction behavior from insufficient knowledge. However, not being “apples-to-apples” indeed may affect cross-model result interpretation.

We address this by calculating AI partners’ MedCOBE score using “all cases” in the evaluation set. We hereby report that AI partners’ ranking consistency between these results and the original MedCOBE score is: Kendall’s τ = 0.91, Spearman’s ρ = 0.98. The results show that AI partners’ relative rankings of MedCOBE scores generally align, regardless of whether they are “apples-to-apples” or not.

We also re-do the correlation analysis from Figure 3 (a) using this all-case MedCOBE score, and the results show that a knowledge-conditioned MedCOBE score correlates better with collaborative acc, supporting our designed knowledge-behavior disentanglement:

MethodSpearman's ρPearson's r
Standalone Acc0.4090.421
MedCOBE Score (All cases)0.7590.614
MedCOBE Score (knowledge-conditioned; as-is)0.8200.677

While we tried to include a “matched-K-is-1-case” experiment, too few cases have all LLMs uniformly reach K_X = 1.0 to support reliable calculation of the MedCOBE score and correlation analysis. Note that each 15-model-matched case is associated with a different combination of 15 LLMs:

Metric20 (All)-model match15-model match
Count (%)1.5645.61

[R4] Response to “human annotation” (W4):

We’d like to mention that the behaviors are labeled by referencing the doctor-annotated explanations in the dataset (line 245). Clinical appropriateness is thus mainly supported by ground truth rather than LLM knowledge.

We additionally recruited 3 specialists to annotate responses from other JAMA domains (60 responses per domain). We report their agreement (κ) with our 3-judge aggregated labels.

SpecialtyAction (κ)Validity (κ)
Ophthalmology0.7480.885
General Internal Medicine0.7270.820
Otolaryngology0.8600.923
Dermatology0.8560.965

Human specialists’ agreement remains fairly high across these new domains. We will incorporate these additional results into the revised manuscript to provide broader evidence for the reliability of both action and validity labeling.


We hope these additional analyses address your concerns about the soundness and generalizability of our evaluation and further strengthen the evidence supporting MedCOBE as a pre-deployment diagnostic sandbox.

Thank you for the careful response and for running the additional experiments during the discussion period. I have no further questions and will maintain my rating of 3.

Reviewer_GifEOpenReview ↗
Rating
2
Confidence
4(+2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedCOBE, a simulation framework used to assess and diagnose the behavioral failures of large language models in clinical human-machine collaboration. The authors point out that traditional medical question-answering benchmarks mainly measure the model's independent diagnostic ability, which fails to fully reflect its performance when interacting with clinical doctors.

Critical strengths
  1. The paper focuses on the difference between the independent diagnostic ability of the model and the actual human-machine collaboration performance, pointing out that high medical question answering accuracy does not necessarily lead to reliable clinical collaboration. This issue is of great significance for the security assessment of medical large-scale models.

  2. The paper evaluated 20 different types of LLMs, using independent evaluation sets and collaborative result test sets, and examined factors such as different simulation roles, clinical confidence, and authority relationships.

Critical weaknesses
  1. The paper relies entirely on LLM simulation of clinical doctors and has not yet been validated through interaction experiments between real doctors and models to determine whether MedCOBE scores can predict actual collaboration performance. Although the author tested different simulators and interaction scenarios, this does not fully demonstrate the consistency between simulated interactions and real clinical decisions.

  2. MedCOBE primarily measures behavioral appropriateness through Affirm and Challenge and categorizes partial disagreement as a Challenge. However, real clinical collaboration often involves uncertainty expression, supplementary examinations, clarification of issues, and appropriate postponement of decisions.

  3. The experiment is mainly based on multiple choice cases from JAMA Clinical Challenge and CasiMedicos Arg, which differ significantly from open diagnosis, complex multi disease coexistence, and dynamic information acquisition in real clinical environments. In addition, cases involving medical imaging use textual descriptions instead of actual images, which cannot verify multimodal collaboration capabilities. Due to the use of publicly available data, the potential impact of exposure to relevant cases during the pre-training phase of the model also needs to be further excluded.

  4. Although the author used three LLM evaluators for majority voting and provided an artificial consistency experiment, the human verification only involved 50 ophthalmology cases, 124 AI responses, and one ophthalmologist. The integrated consistency coefficient of behavior classification was 0.748, which was even slightly lower than the 0.770 of one of the individual evaluators. The lack of validation across different clinical specialties, multiple experts' consistency, and complex interaction scenarios leaves the reliability of the behavior score in question.

  5. The core methods mainly combine existing LLM simulations, behavior classification, statistical aggregation, clustering, and prompt interventions, lacking deeper methodological innovation. Although the appendix provides behavioral profiling ablation experiments and repeated interaction analysis, model level interventions mainly focus on two LLMs, which is not sufficient to demonstrate the stable effects of the method on different models and clinical tasks.

  6. Core correlation analysis was based on 20 models and 100 independent cases. The authors have provided statistical support through Bootstrap, leave-one-out model analysis, and partial correlation tests, but these experiments are still limited to the same simulation evaluation system and do not verify the predictive ability in independent hospital data or real user groups. In addition, the main comparison object is independent question and answer accuracy, and there is a lack of direct comparison with existing multi-round interactive evaluations and other behavioral scoring methods.

Questions for authors
  1. Can you provide at least one independent interactive experiment involving real clinical doctors to verify the correlation between MedCOBE scores and actual collaborative performance? Can the simulator accurately reflect doctors' acceptance and rejection behaviors towards AI recommendations?

  2. Can it be verified that the current assumptions hidden by the simulator are consistent with the opinions actually expressed in its natural language? How to ensure that models are not erroneously penalized for failing to respond to hidden assumptions when clinicians do not express their position explicitly?

  3. Can the intervention effect be validated on more models, different clinical tasks, and independent datasets, and further demonstrate that the performance improvement does indeed come from the model specific behavior provided by MedCOBE by controlling the Critic model.

7 comments
AuthorsOpenReview ↗

To Reviewer GifE (1/2)

We appreciate your time and effort. Your constructive comments have provided many insights for improvement!


[R1] Response to “adding experiments with real doctors” (W1 & Q1):

MedCOBE aims to go beyond exam-like benchmarks and provide a controllable, scalable diagnosis of LLM partners, rather than directly predicting real-world outcomes (Limitations; Appendix B).

Moreover, plotting correlations with clinicians requires multiple data points (multiple clinicians * multiple LLMs), which involves substantial time and budget. Instead, we focused on improving task realism by simulating diverse usage contexts in Table 2. Also, in [R3] below, we further present two additional experiments for this purpose: (i) open-ended diagnosis and (ii) not providing full case information up front.

We’d also like to clarify that the simulator is not intended to replicate human clinicians' behavior. Rather, it provides controlled interaction contexts to assess how AI partners collaborate with users.


[R2] Response to “Affirm and Challenge” (W2):

As noted in Limitations, we agree that real-world interactions involve more nuanced behaviors. We intentionally adopt a simplified formulation to focus on two safety-relevant failure modes: sycophancy toward incorrect hypotheses and obstruction of correct ones. This bi-directional design enables systematic, interpretable behavioral diagnosis and provides actionable insights for mitigation.


[R3] Response to “Dataset format and publicity” (W3):

To address your concern about the difference between data format and real-world scenarios, we additionally evaluate (i) open-ended diagnosis and (ii) “without case information up front”. MedCOBE remains effective in both settings:

MethodSpearman's ρPearson's r
Standalone Acc0.4090.421
MedCOBE (Open-ended)0.7160.593
MedCOBE (w/o Case Information up front)0.7000.602

To better isolate the effect of interaction behavior, we used expert-annotated image captions from JAMA rather than relying on VLMs to interpret the images. In the revised manuscript, we will explicitly mention that our results do not guarantee transfer to multimodal settings.

Also, we agree with your concern about data exposure. However, our main focus is whether standalone knowledge carries over to interaction via proper behavior, regardless of how the knowledge was obtained. Thus, identifying the knowledge source is not essential to our primary analysis, although we cannot guarantee that it doesn’t influence the observed behavior. We will discuss this in Limitations.


[R4] Response to “behavior labels” (W4):

Regarding majority voting, we use it to avoid bias from any single judge, not to guarantee higher agreement with human annotations.

We additionally recruited specialists to annotate responses from other JAMA domains (60 responses per domain). We report their agreement (κ) with our 3-judge aggregated labels.

SpecialtyAction (κ)Validity (κ)
Ophthalmology0.7480.885
General Internal Medicine0.7270.820
Otolaryngology0.8600.923
Dermatology0.8560.965

Human specialists’ agreement remains fairly high across these new domains.


[R5] Clarification to “innovation” (W5):

We agree that MedCOBE builds on established techniques. However, as you noted in Strengths, our scientific contributions go beyond methodological design: We demonstrate that knowledge does not guarantee successful collaboration We show that collaboration failures are directional (tendencies to sycophancy and obstruction), not merely single-value scalar results. We propose behavioral diagnosis that not only pre-surfaces such failures but also enables targeted mitigation. None of these can be achieved by standalone accuracy on common exam-like benchmarks.

AuthorsOpenReview ↗

To Reviewer GifE (2/2)

[R6] Response to “additional intervention experiments” (W5 & Q3)

To isolate the effect of model-specific profiles in critic intervention, we run an experiment in which the critic is given different profiles. The results suggest the effect of MedCOBE-guided intervention:

AI PartnerMedCOBE-guided Critic (Original Profiles)Critic w/ Switched Profiles
GPT-5.6-Sol0.8530.833
Claude-Opus-50.7930.777

In the paper (Table 10), we also presented results for switched-profile model-level intervention.

Regarding generalization to other datasets, we present the result below. Importantly, we re-use the profiles from our current paper, as MedMCQA does not provide expert explanations for us to label interaction behavior. This further demonstrates the generalization of MedCOBE-diagnosed profiles:

AI Partner+ Naive Critic+ MedCOBE-guided Critic
GPT-5.6-Sol80.3382.50
Claude-Opus-581.3382.67

Regarding generalization to other LLMs, we add experiments with XX other LLMs, and the results show that MedCOBE-guided intervention remains effective:

AI Partner+ Naive Critic+ MedCOBE-guided Critic
GPT-5.6-Sol0.8170.853
Claude-Opus-50.7530.793
Gemini-3.1-Pro0.8230.857
GLM-5.30.7170.857
LLaMA-3.1-8B0.6300.670

[R7] Response to “generalization” and “comparing with existing evaluation” (W6):

We hereby compare MedCOBE with counterfactual stress-testing, which is commonly used to assess LLM sycophancy. This method deliberately presents the AI partner with correct or incorrect clinical statements and assesses whether it affirms or challenges them appropriately. We present the correlation analysis (same as Figure 3 (a)):

MethodSpearman's ρPearson's r
Single-turn counterfactual test0.7470.612
MedCOBE0.8200.677

We presume MedCOBE’s win stems from evaluating LLMs throughout the natural progression of interaction rather than single-value accuracy.

Regarding cross-data generalization, since it’s non-trivial to acquire real hospital data, we instead present a correlation analysis between (i) the MedCOBE score calculated in our paper and (ii) the collaborative acc in an out-of-domain (OOD) dataset (MedMCQA):

DatasetSpearman's ρPearson's r
The Out-of-domain MedMCQA dataset0.6700.743
The current held-out outcome set0.8200.677

Despite being OOD, the correlation remains substantial, suggesting the generalization of the MedCOBE sandbox.


[R8] Response to “alignment between implied hypotheses and NL responses” (Q2):

We address this alignment by prompting the simulator to explicitly generate its hypothesis at the end of its response (the prompt is in Appendix). This hypothesis label is not visible to the AI partner but only to the post-hoc behavior judge when measuring the MedCOBE score. We will move this implementation detail to Section 3.

Regardless, we agree that it does not fully guarantee the alignment. We address this concern by presenting additional verification of whether LLMs (AI Partners) can correctly infer the user hypothesis from the NL utterance alone. We report the average accuracy (%):

DatasetLLM AccuracyRandom
JAMA Challenge (4-class)94.225.0
CasiMedicos-arg (5-class)83.020.0

Thank you for the response and additional experiments. However, several important issues remain.

  1. MedCOBE still relies entirely on simulated interactions. The clarification of the benchmark is helpful, but its clinical relevance remains uncertain.

  2. The reported hypothesis inference accuracy of 83.0% on CasiMedicos-arg indicates that a substantial proportion of user intentions may be misinterpreted. Still not sure whether models are unfairly penalized when the hidden hypothesis differs from natural-language expression.

  3. The switched-profile experiments show relatively small differences, without confidence intervals or significance testing.

  4. MedMCQA remains a public, exam-style dataset, and the reported correlations are still based on simulated collaboration. Its statistical significance and practical importance remain unclear. Potential data pollution and the limited novelty are also insufficiently addressed.

Reviewer_7J8qOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether an LLM that answers clinical decision questions correctly in a standalone, exam-style setting still supports a clinician appropriately once a second decision-maker enters the loop, and reports a substantial degradation in this setting ("lost in collaboration"). To study this at scale the authors build a simulated sandbox: real clinical decision cases, an LLM clinician simulator that opens with either a correct or an incorrect working hypothesis, turn-level labelling of the partner's reply as Affirm / Challenge / Other by LLM judges, two recall-style metrics aggregated into one score on cases where the model is considered to have sufficient knowledge, 20 LLMs evaluated, behavioural profiles that separate sycophancy from obstruction, and two prompt-level interventions.

Critical strengths
  • The phenomenon is worth reporting. The gap between standalone accuracy and performance in a deployment-like interaction is real, and case-level examples where a model demonstrably knows the answer yet fails to defend it are a useful addition to the clinical-evaluation literature.
  • The analysis section is where the paper has value. Rather than stopping at scalar scores, it characterizes each model's disposition — sycophancy toward incorrect clinician hypotheses versus obstruction of correct ones — and shows that models with comparable scores fail in different directions. This qualitative, direction-aware diagnosis is the creative part of the paper, and it is the part I would keep.
Critical weaknesses

W1. Behaviour labels come from LLM judges aggregated by majority vote, with no professional annotation at a scale that could support the conclusions. The metric, the 20-model ranking and the behavioural profiles all rest on labels produced by three LLM judges, and the only expert check is 124 responses from one ophthalmologist. Majority voting across same-generation models reduces variance but not shared bias, and agreement among judges is not evidence of clinical appropriateness.

W2. The methodological novelty is compositional. The framework is red teaming (turn-level stance labels assigned by LLM judges) combined with counterfactual probing (correct versus incorrect initial hypothesis).

W3. There is no theoretical contribution, and the contribution as it stands does not meet the main-conference bar. The paper builds an instrument and reports correlations; it offers no account of why collaboration degrades, and no claim that could be falsified independently of the score it proposes.

W4. The claimed mechanism for the naive critic is unsupported and contradicted by the evidence cited. Section 5 attributes the naive critic's accuracy gain to knowledge provision rather than behavioural guidance, citing higher S_Lost. But S_Lost is defined only on the K_X = 1.0 stratum, so any gain from supplying information on cases the model cannot solve would fall outside that denominator, while a rise in S_Lost indicates harm precisely on cases the model can already solve. Moreover the comparison cited is against the guided critic, not against vanilla. Ask: critic accuracy, partner compliance rate, and accuracy broken down by knowledge stratum, plus a placebo critic.

Questions for authors
  1. Positioning against red teaming and counterfactual evaluation. Section 3.2's correct/incorrect initial-hypothesis design is a counterfactual manipulation, and the turn-level Affirm/Challenge judging in Section 3.3 is a standard red-team rubric. What does the framework provide that a scripted counterfactual probe (present an incorrect hypothesis in a single turn, measure whether the model pushes back) does not? Please give one experiment that distinguishes the two.

  2. Separating information, knowledge and prompt perturbation in the critic. Appendix F states that neither critic variant receives the ground truth, the expert explanation, earlier dialogue, or the AI draft, and that neither may supply a rationale or an alternative answer; the naive critic returns a single 1/0/not-assessable verdict rendered as one fixed sentence plus "This assessment may be mistaken." What the critic transmits is therefore a verdict on the clinician's conclusion, not medical knowledge. Please report (i) the critic's own accuracy against the ground truth, (ii) the partner's compliance rate conditional on a verdict of 0 or 1, (iii) collaborative accuracy and S_Lost stratified by knowledge (K_X = 1.0, 0.5–0.9, 0), and (iv) a placebo critic that emits the same phrasing with a random verdict. If the placebo also improves accuracy, the effect is prompt perturbation rather than any information channel.

3 comments
AuthorsOpenReview ↗

To Reviewer 7J8q (1/2)

We appreciate your time and effort. We believe addressing your critical feedback will help us improve our work.


[R1] Response to “methodological novelty” (W2 & Q1):

We agree that MedCOBE builds on established techniques. However, as you noted in Strengths, our scientific contributions go beyond methodological design: We demonstrate that knowledge does not guarantee successful collaboration We show that collaboration failures are directional (tendencies to sycophancy and obstruction), not merely single-value results. We propose behavioral diagnosis that not only pre-surfaces such failures but also enables targeted mitigation. None of these can be achieved by standalone accuracy on common exam-like benchmarks. Compared with the suggested single-turn counterfactual test, MedCOBE shows stronger correlations with collaborative outcomes, potentially because it captures behavior across evolving interactions rather than static single-turn signals:

MethodSpearman's ρPearson's r
Single-turn counterfactual test0.7470.612
MedCOBE0.8200.677

[R2] Clarification to “why collaboration degrades” (W3):

MedCOBE offers a behavioral diagnosis of collaboration failures: models with sufficient clinical knowledge can still fail through their tendencies toward sycophancy or obstruction. This helps explain why standalone performance does not necessarily translate into effective collaboration.


[R3] Response to “appropriateness of labels” (W1):

We’d like to mention that the behaviors are labeled by referencing the doctor-annotated explanations in the dataset (line 245). Clinical appropriateness is thus mainly supported by ground truth rather than LLM knowledge.

We additionally recruited 3 specialists to annotate responses from other JAMA domains (60 AI partner responses per domain). We report their agreement (κ) with our 3-judge aggregated labels.

SpecialtyAction (κ)Validity (κ)
Ophthalmology0.7480.885
General Internal Medicine0.7270.820
Otolaryngology0.8600.923
Dermatology0.8560.965

Human specialists’ agreement remains fairly high across these new domains.


[R4] Response to “not falsifiable” (W3):

Our central claims are falsifiable through outcomes measured independently of the proposed MedCOBE score: The lost in collaboration phenomenon is established by comparing standalone and collaborative decision outcomes, without relying on the MedCOBE score. We evaluate the MedCOBE score's predictive power against collaborative Acc on a separate held-out case set. If the score failed to predict collaborative Acc better than standalone Acc does, its utility would not be supported. The utility of diagnosed behavioral profiles is supported by the intervention experiments against non-MedCOBE-guided intervention (Figure 5 & 6). Their falsifiability is supported by Table 10, showing that mismatched profiles yield suboptimal results.


[R5] Response to “statement regarding naive critic”(W4):

We appreciate your careful identification of our misphrased statement. The naive critic indeed provides an external correctness signal about the hypothesis. We will rewrite the statement to avoid attributing the observed gains to a specific mechanism without sufficient evidence.

Second, we will also distinguish vanilla-to-naive from naive-to-guided comparisons in our discussion of Figures 5 and 6.

AuthorsOpenReview ↗

To Reviewer 7J8q (2/2)

[R6] Response to “additional experiments regarding critic”(Q2):

Further verifying the critic's effect can help readers better identify our contribution. We present the requested experimental results below


(i) The critic's own acc against the ground truth:

Critic's LLMsAccuracy
GPT-5.6-Sol0.74
Claude-Opus-50.74
Gemini-3.1-pro0.75

(ii) Partner's compliance rate conditional on critic-given verdict (avg of three critic LLMs):

AI PartnerVerdict 0: NaiveVerdict 0: MedCOBEVerdict 1: NaiveVerdict 1: MedCOBE
GPT-5.6-Sol67.6%87.5%93.9%97.2%
Claude-Opus-585.3%100.0%89.6%96.8%

MedCOBE-guided critics achieve higher verdict-compliance rates than naive critics.


(iii) Collaborative Acc and S_lost stratified by knowledge under critic intervention (avg of three critic LLMs):

Because S_lost is, by design, calculated based on K_X =1 cases to quantify lost in collaboration, we present only the stratified collaborative acc under critic intervention. The results show that MedCOBE-guided Critic did not improve collaborative acc in cases with K_X = 0, suggesting that prompt perturbation does not explain the improvements reported in the paper.

AI PartnerK_X RangeCase Count per RangeNaive CriticMedCOBE-guided
GPT-5.6-Sol02373.91% (+4.35pp)69.57% (+0.00pp)
GPT-5.6-Sol0.1–0.9650.00% (+33.33pp)66.67% (+50.00pp)
GPT-5.6-Sol1.07187.32% (-1.41pp)92.96% (+4.23pp)
Claude-Opus-502373.91% (+4.35pp)69.57% (+0.00pp)
Claude-Opus-50.1–0.91020.00% (-20.00pp)70.00% (+30.00pp)
Claude-Opus-51.06779.10% (+10.45pp)82.09% (+13.43pp)

(iv) Placebo MedCOBE-guided critic with same phrasing and a random verdict (avg of three critic LLMs): The much smaller performance changes from a random verdict suggest that the result reported in the paper is not due to prompt perturbation.

AI PartnerRandom VerdictOriginal Verdict (In-Paper)
GPT-5.6-Sol-0.3%+5.3%
Claude-Opus-5+2.2%+13.3%
Reviewer_7J8qOpenReview ↗

Score increased after reading the additional response

Thank you for the explanation! I increased my score to 3 after reading the response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of additional experiments and key clarifications

Dear AC and reviewers,

We sincerely thank the AC for managing the review and appreciate the reviewers' engagement and their consideration of our rebuttal. We summarize our rebuttal responses so far, with references to the corresponding reviewers' concerns.


Additional Experiments

  1. Extended human expert validation (7J8q W1; GifE W4; VuZm W4): Supports the reliability of our post-hoc behavior judgments across clinical specialties.
  2. Compare with counterfactual stress-testing (7J8q W2/Q1; GifE W6): Shows the benefit of MedCOBE over common single-turn counterfactual testing.
  3. Include more task formats (Open-ended/incomplete information) (GifE W1/W3/Q1; VuZm W2): Supports MedCOBE's effectiveness across other deployment-like task formats.
  4. OOD generalization (GifE W6 and 2nd-round comment): Supports MedCOBE's ability to predict collaborative outcomes on different datasets.
  5. Knowledge-conditioning ablation (VuZm W3): Shows consistent model rankings even in knowledge-unconditioned settings, and highlights the benefit of knowledge conditioning (original setting).
  6. Requested critic experiments (7J8q W4/Q2): Supports that the improvements are most likely not merely due to prompt perturbation.
  7. Intervention generalization (GifE W5/Q3): Supports the effectiveness of MedCOBE-guided interventions across LLMs and datasets.
  8. Model-specific profile ablation (GifE W5/Q3 and 2nd-round comment): Highlights the benefit of using model-specific behavioral profiles in the critic and discusses its performance gap compared to the switched-profile critic.
  9. Add Human and LLM accuracy in Hypothesis–utterance alignment (GifE Q2): Supports the alignment between clinician hypotheses and natural-language responses.

Key Clarifications

  1. Novelty and falsifiability (7J8q W2/W3; GifE W5): Clarified our scientific contributions and independently falsifiable claims.
  2. Action type (GifE W2; VuZm W1): Explained our focus on Affirm/Challenge and scenario-aware validity judgments.
  3. Critic's explanation (7J8q W4): Corrected our explanation of the naive critic's improvements.
  4. Clinical applicability (GifE W1/Q1; VuZm W2): Clarified MedCOBE's role as a pre-deployment diagnostic sandbox rather than a substitute for real-clinician evaluation.
  5. Dataset limitations (GifEW3): Clarified the limitations regarding data exposure and multimodal settings.

We believe these strengthen the paper and cover the reviewers' major concerns.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.