acceptodds
Under review as a conference paper at ICLR 2027

OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

Abstract

A multidisciplinary tumor board meeting serves as a critical decision point along the cancer patient journey, particularly representing one of the last hopes when standard options are exhausted. Agentic biomedical reasoning in large language models (LLMs) holds promise for timely multidisciplinary cancer decision-making. However, the development of these models is fundamentally constrained by the lack of evaluation using patient cases brought before real-world tumor boards, with discussion trajectories that integrate multimodal clinical observations and longitudinal patient timelines. We introduce OpenTumorBoard, a real-world benchmark at an unprecedented scale, with 611 patient cases and 19,157 turns of cancer discussions across ten specialist roles, transcribed from 12,534 minutes of video recordings on YouTube. The benchmark evaluates (1) LLMs joining a single Specialist Turn in the real discussion by responding to a clinically significant question; (2) Board Simulation of an entire back-and-forth discussion to reach a consensus on therapy recommendations, surgical plans, next actions and clinical trial matching. Extensive evaluation of 14 general-purpose frontier models and medical LLMs identifies substantial limitations. The best-performing models reach only 3.43 out of 5 in clinical equivalence to actual specialist answers and 2.78 out of 5 in alignment with true tumor board conclusions. The best-performing models struggle both to answer naturally occurring questions posed by specialists and to orchestrate coherent multidisciplinary discussion trajectories. Supervised finetuning and reinforcement learning yield substantial improvements on a held-out test set, suggesting that OpenTumorBoard can serve as a training ground that effectively adapts LLMs with optimized clinical discussions. To confirm the quality of the benchmark, we recruit three M.D. experts and verify high information coverage and factuality of patient cases, as well as strong fidelity of the final consensus. We will release both OpenTumorBoard and its automated curation pipeline to enable scalable development and evaluation of LLMs for multidisciplinary, personalized cancer decision-making.

Primary area:
datasets and benchmarks
Keywords:
benchmark, large language models, clinical decision-making, multimodal reasoning, medical AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sdABOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OpenTumorBoard benchmarks multidisciplinary tumor board decision-making using 611 multimodal patient cases from public YouTube recordings. Its two tasks, SPECIALIST TURN and BOARD SIMULATION, evaluate specialist responses and consensus formation. Evaluations of 14 LLMs reveal substantial performance gaps, with supervised finetuning and reinforcement learning yielding modest gains.

Critical strengths
  1. OpenTumorBoard captures real tumor board discussions across ten specialist roles and combines multimodal clinical evidence with longitudinal patient histories. Its two tasks evaluate specialist responses and board-level consensus, closely reflecting real clinical workflows.

  2. The paper presents a clear automated pipeline for transcription, role identification, slide extraction, and temporal alignment. Expert audits and component-level checks assess data quality and pipeline reliability.

Critical weaknesses
  1. The curation pipeline relies heavily on LLMs for role identification, case segmentation, and question and conclusion extraction. Errors in these steps could propagate through the dataset, and the human audits cover only a limited sample.

  2. The RL experiments use LLM judges to provide rewards and measure improvements. This setup may favor responses that satisfy the judge without improving clinical quality. More human evaluation would help establish whether the reported gains reflect meaningful improvements.

  3. The main evaluation relies on a single judge model, Qwen3.8-27B, which may influence model rankings. The clinician evaluation covers only 15 cases, and the paper does not report the correlation between clinician ratings and judge scores.

  4. The cases come from publicly available YouTube recordings and may differ systematically from routine clinical cases. The paper provides limited evidence that the benchmark represents different institutions, regions, and tumor board formats.

Questions for authors
  1. What slide content will you release: captions and source URLs with timestamps, or extracted slide images as well? If images are not redistributed, how can others reproduce the multimodal benchmark at scale?

  2. How did you quantitatively validate the removal of information leakage? Could you report key statistics in the main text, such as the proportion of turns or frames removed, and checks confirming that the inputs do not reveal the final conclusions?

  3. Did you evaluate the systems using multiple independent LLM judges? Could you report rank correlations and the stability of system rankings across judges, or add this analysis if it has not been conducted?

  4. Could you evaluate the RL improvements using a different judge family and a small clinician-rated subset? This would help assess whether the gains reflect better clinical responses or overfitting to the reward judge.

  5. How representative are the YouTube-sourced tumor boards in terms of geography, language, academic versus community settings, and case acuity? Could you quantify the concentration of cases by institution and cancer type, and discuss plans to include data from additional institutions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sdAB (1/3): Curation quality (W1) and RL robustness (W2)

Response to W1: Thank you for raising the concern that errors in the LLM-based curation steps could propagate through the dataset. We extended the Appendix A audit from 15 recordings to all 219, and clinicians reviewed the case summaries and reference conclusions of 10 additional test cases beyond those in Section 3.4.

We used the original recordings as the reference for checking each curation step. Across all recordings, we checked every speaker role, case boundary, and extracted slide frame, along with a random sample of 811 (5%) question and answer rewrites. The audit found that 97.8% of speaker roles were correct, 99.2% of case boundaries were correct or differed only slightly, and 99.5% of extracted frames were slides. Among the sampled rewrites, 0.7% changed the question's meaning, and all preserved the answer's meaning. Because errors are rare at every step, few can propagate into the model inputs and references. In addition, all 16,215 Specialist Turn reference answers retain the clinician's verbatim wording and can be checked against the source utterance. In the clinical review of 447 pairs from the 15 cases in Section 3.4, three clinicians found that 94.4% of answers were fully supported by the recorded discussion and 92.6% were clinically correct. Following the reviewer's suggestion, we asked clinical experts to annotate 10 additional test cases, expanding our clinician-verified subset to 25 cases in total. In this newly annotated batch, all 10 reference conclusions (100%) faithfully captured the tumor board's final decisions, and 9 of 10 case summaries (90%) preserved the essential clinical information of the case. These outcomes closely mirror our findings on the initial 15 cases, confirming the stability and consistency of our dataset construction. Given that scaling to the entire dataset would require roughly 4,900 clinician-hours, this 25-case audit offers a practical and reliable validation of overall extraction quality, and we will incorporate these updated results into the revised manuscript. Clinicians have also scored model outputs on 30 test cases, including these 10, while blinded to model identity, as described in our response to W2.

Response to W2: We thank the reviewer for raising the question of whether the RL gains extend beyond the reward judge. We tested this with an evaluation judge from a different model family, with a different reward judge, and with blinded clinician scoring of the RL outputs.

We first rescored the base, SFT, and RL models on Board Simulation with GPT-6.1 Sol, which belongs to a different model family from both the RL reward judge (Qwen3.6-27B) and the paper's evaluation judge (Qwen3.8-27B). The ordering was unchanged, with the base model at 1.64, SFT at 1.74, and RL at 1.82. We then retrained RL with Gemma 4 31B as the reward judge instead of Qwen3.6-27B. The final checkpoint scored 1.87 under the paper's judge and 1.79 under GPT-6.1 Sol, compared with 1.86 and 1.82 for the original RL model, and it remained above SFT and the base model under both judges.

To assess whether these gains reflect better clinical responses, clinicians newly scored the base, SFT, and RL conclusions on 30 test cases with the conclusion alignment rubric, blinded to model identity and mixed with outputs from three other models. The clinicians gave the base, SFT, and RL models 1.93, 2.33, and 2.70, the same ordering as the judges, and scored RL above the base model in 14 cases and below it in only 5. They also scored RL (2.70) close to DeepSeek-V4-Pro (2.73) and Gemini 3.7 Flash (2.80) and above MedGemma 27B (2.60).

AuthorsOpenReview ↗

Response to Reviewer sdAB (2/3): Judge robustness (W3) and representativeness (W4)

Response to W3: We thank the reviewer for the suggestion to examine how much the rankings depend on a single judge and how judge scores relate to clinician ratings. We rescored model outputs with two judges from other model families, Claude Opus 5.5 and GPT-6.1 Sol, using the same rubric and references as the original judge.

On Board Simulation, both judges rescored all outputs, and their model rankings correlated with the original judge's at Spearman ρ = 0.977 (Claude Opus 5.5) and 0.956 (GPT-6.1 Sol). Gemini 3.7 Flash ranked first under all three judges. On Specialist Turn, where the two judges rescored every model's responses to two questions per test case, the rankings correlated at ρ = 0.983 to 1.00, quadratic-weighted kappa between judges on individual responses ranged from 0.865 to 0.909, and all three judges agreed on the order of every pair of adjacent models.

To measure the correlation between clinician ratings and judge scores, clinicians newly scored 180 Board Simulation conclusions from 30 test cases with the judge's rubric while blinded to model identity, and the expanded clinician review described in our response to W1 goes beyond the original 15 cases. The clinicians' ranking of the models correlated with the judge's at Spearman ρ = 0.94, and 90.6% of their scores on individual conclusions were within one point of the judge's.

Response to W4: We thank the reviewer for asking how representative the recordings are. To quantify this along the dimensions the reviewer lists, we characterized all 611 cases by geography, language, hosting setting, acuity, channel, and cancer site, using YouTube channel metadata and our case annotations.

By geography, 34.0% of cases come from the United States, 16.9% from international multi-country boards, and 16.4% from other countries, while 32.7% could not be determined from the metadata. Speech is in English in 215 of the 219 recordings. By hosting setting, 38.1% of cases come from professional societies and conferences, 24.5% from patient advocacy organizations, 17.7% from medical education media, 2.9% from academic medical centers, and 1.0% from community oncology, with 15.7% other or undetermined. By acuity, 38.0% of cases are metastatic, 26.8% localized, 18.7% locally advanced, and 12.8% recurrent. The cases are spread across many sources: the largest of the 62 channels holds 17.0% of cases (Herfindahl index 0.076, equivalent to about 13 equally sized channels), and the largest of the 17 cancer sites, thyroid and parathyroid, accounts for 20.6%.

These boards are broadcast for education and likely favor complex cases, which is the setting where multidisciplinary review matters most, and we will state this in the limitations. Routine tumor boards are rarely shared because of patient privacy, so public recordings are currently the main source that can be openly released. To include more institutions, our released pipeline allows any institution to add its own recordings, and we welcome such contributions.

AuthorsOpenReview ↗

Response to Reviewer sdAB (3/3): Questions Q1–Q5

Response to Q1: We will release each slide's caption, source YouTube URL, and timestamp, together with the slide-to-case index, the data splits, and the full curation pipeline. With these, others can regenerate the slide images from the public recordings and reproduce the multimodal benchmark at scale.

Response to Q2: We validated leakage removal by measuring how much of each input was removed and by checking whether the edited inputs still reveal the decision. De-leaking edits only the model inputs and never the references. Before de-leaking, a classifier confirmed post-decision information in 58 of the 184 test cases (31.5%). De-leaking changed the Board Simulation inputs for 88 test cases (47.8%), removing 9.5% of input text tokens and 94 of 1,435 slides (6.6%), and removed 149 of 4,993 Specialist Turn test questions (3.0%). To check whether the text inputs still reveal the decision, two LLMs not used in curation, DeepSeek-V4-Pro and Gemini 3.7 Flash, judged whether each case's inputs state or imply the final conclusion. They flagged 80 and 79 cases before de-leaking and only 16 and 13 after. Excluding the 20 cases flagged by either model after de-leaking moves no configuration by more than one rank (ρ = 0.991). We will report these statistics in Section 2.2.

Response to Q3: Yes. In addition to Qwen3.8-27B, we rescored model outputs with Claude Opus 5.5 and GPT-6.1 Sol. The rank correlations and the stability of the rankings across the three judges are reported in our response to W3.

Response to Q4: Yes. As described in our response to W2, the ordering of the base, SFT, and RL models is unchanged under GPT-6.1 Sol, a judge from a different model family, and RL retrained with Gemma 4 31B as the reward judge also remains above SFT and the base model. Clinicians blinded to model identity ranked the three models in the same order, scoring the base, SFT, and RL models 1.93, 2.33, and 2.70 on 30 test cases.

Response to Q5: Please see our response to W4, which reports the distribution of cases by geography, language, hosting setting, and acuity, the concentration of cases by channel and cancer site, and our plan for including data from additional institutions.

Thank you for the detailed responses and additional experiments. These have satisfactorily addressed my main concerns. I look forward to seeing the additional results and clarified limitations incorporated into the revised manuscript.

Reviewer_sfvwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenTumorBoard, a benchmark built from 611 real-world cancer cases and 19,157 discussion turns extracted from recorded multidisciplinary tumor board meetings. It evaluates two tasks: Specialist Turn, where models answer clinical questions as designated specialists, and Board Simulation, where models generate multidisciplinary discussions and final treatment recommendations. Experiments across 14 models reveal substantial gaps in clinical reasoning and alignment with real tumor board decisions. The authors also show that SFT and RL on recorded discussion trajectories can improve conclusion alignment.

Critical strengths
  1. Clinically meaningful problem. The paper addresses an important real-world problem: evaluating LLMs' ability to support multidisciplinary cancer decision-making, which has substantial practical value.

  2. Broad and valuable benchmark coverage. The benchmark covers 611 real-world patient cases, multiple clinical modalities, 10 specialist roles, and complete discussion trajectories. This provides a valuable resource for evaluating both individual specialist responses and multidisciplinary decision-making.

Critical weaknesses
  1. The role and effectiveness of simulated discussions remain unclear. The paper does not compare discussion simulation against directly generating the final conclusion or using a multi-agent framework where separate agents represent different specialists and ground their discussions in clinical slides and evidence. More importantly, the intermediate discussions are not evaluated. It remains unclear whether models can accurately simulate different specialist roles, whether their intermediate clinical judgments are correct, and how performance varies across roles.

  2. Insufficient evaluation rigor and reliability analysis. The paper relies heavily on LLM-as-judge evaluation without validating its agreement with clinician assessments of model outputs. It also lacks repeated inference runs and multi-seed experiments to assess consistency and robustness. Statistical significance is insufficiently reported, making it difficult to assess the reliability of the observed performance differences and training improvements.

Questions for authors
  1. Effectiveness of discussion simulation: Is there a comparison with directly generating the final conclusion without simulating discussions? Have you explored using independent specialist agents that ground their discussions in the clinical evidence? These comparisons would help clarify the actual benefit of discussion simulation.

  2. Quality of intermediate discussions: Have you evaluated the clinical correctness of the generated discussion turns, beyond the final conclusion? In particular, can models accurately simulate different specialist roles, and does their performance vary across roles?

  3. Evaluation reliability: Have you assessed the agreement between the LLM judge and clinician evaluations of model-generated outputs? Additionally, are repeated inference runs, multiple training seeds, or statistical significance results (e.g., p-values or confidence intervals) available to establish the reliability of the reported improvements?

7 comments
AuthorsOpenReview ↗

Response to Reviewer sfvw (1/3): Baselines and intermediate discussion evaluation (W1)

Response to W1: We thank the reviewer for suggesting direct conclusion baselines, independent specialist agents, and intermediate trajectory evaluations to clarify the role of simulated discussions. Holding the model and clinical inputs fixed, we evaluated 12 of the 14 configurations with a conclusion-only prompt. Simulating the discussion improves alignment for leading models including Grok 4.6 (+0.23), Gemma 4 31B (+0.14), and Gemini 3.7 Flash (+0.11), exceeding run-to-run variation (up to 0.09), while other models perform comparably to direct generation (average +0.026, within variation). Furthermore, the recorded discussions provide valuable training supervision, as reinforcement learning fine-tuning on these trajectories raises Board Simulation alignment by 17.7% (from 1.58 to 1.86).

To test separate specialist agents grounded in the clinical evidence while holding the base model (MedGemma 27B) fixed, we built a multi-agent baseline where a moderator selects three to six of ten specialist roles to independently assess the summary and slides, respond to one another, and inform a moderator conclusion. This framework scores 2.13, matching our single-model simulation (2.15) within run-to-run variation and showing that single-model simulation captures multi-perspective deliberation just as effectively without the overhead of separate agent instances. To directly evaluate intermediate discussions and specialist role fidelity, we first scored simulated discussions against recorded ones across six dimensions (fidelity to case facts, clinical reasoning, coverage, specialist contribution, deliberation, and conclusion grounding).

These metrics clearly separate models, with clinical reasoning ranging from 1.91 (MedReason-8B) to 4.95 (DeepSeek-V4-Flash), specialist contribution from 1.81 (MedReason-8B) to 4.99 (Claude Opus 5), and invented case facts occurring in 0% to 58% of discussions. Rankings remain consistent under Claude Opus 5.5 (ρ = 0.89 to 0.99 across the first five dimensions), and discussion quality strongly tracks conclusion alignment under both judges (ρ = 0.70 to 0.81 and 0.71 to 0.79). Second, evaluating Specialist Turn performance across nine models shows mean clinical equivalence (and critical-error rates) ranging from 2.85 (15.6%) for surgeons, 2.89 (17.4%) for molecular pathologists, and 2.92 (16.0%) for medical oncologists to 3.00 (13.1%) for radiation oncologists, 3.06 (12.8%) for pathologists, and 3.24 (10.2%) for radiologists. Model rankings are highly consistent across roles (Spearman ρ = 0.93 to 1.00 against the overall ranking, with DeepSeek-V4-Pro strongest and MedReason-8B weakest in every major role), and cross-role variation (2.85 to 3.24) is much smaller than cross-model variation (2.09 to 3.43), confirming that performance depends primarily on underlying model capability rather than the assigned specialty. We will add both baselines and these trajectory and role evaluations to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer sfvw (2/3): Evaluation reliability (W2) and discussion baselines (Q1)

Response to W2: We thank the reviewer for emphasizing evaluation rigor, multi-run consistency, and statistical significance. To verify inference stability while holding the task and prompts fixed, we reran a subset of sampled configurations three to four times. On Board Simulation, the standard deviation ranges from 0.009 to 0.039 and most adjacent rank gaps are 3 to 18 times larger, leaving only two near-tied configurations within run-to-run variation, while on Specialist Turn the standard deviation is 0.004 to 0.009 and the smallest adjacent gap is seven times the run-to-run range, with greedy decoding producing identical outputs across every run. To confirm that our training improvements are not specific to a single random seed, we trained two additional reinforcement learning seeds that achieve 1.79 and 1.75 alongside the paper run of 1.86 (mean 1.80), with every seed outperforming both supervised fine-tuning (1.67) and the base model (1.58).

To establish statistical significance across repeated generations and judge evaluations, we conducted Welch's t-tests over at least four complete runs per model with Benjamini-Hochberg correction, requiring the same directional sign across all three judges (Qwen3.8-27B, Claude Opus 5.5, and GPT-6.1 Sol). Under these criteria, 35 of 36 Specialist Turn model comparisons (including every adjacent rank) and 78 of 91 Board Simulation comparisons are statistically separated, with Gemini 3.7 Flash significantly outperforming every other configuration (t = 13.1, q less than 0.001), and we will explicitly report unseparated configurations as statistical ties. Both training gains are likewise statistically significant, as supervised fine-tuning improves over the base model by 0.09 (t = 4.7, q = 0.004) and reinforcement learning improves over supervised fine-tuning by a further 0.21 (t = 10.6, q less than 0.001), maintaining the same direction under GPT-6.1 Sol.

Regarding alignment with clinician assessments, clinicians newly scored 180 Board Simulation conclusions from 30 test cases, produced by six models, using the exact judge rubric while blinded to model identity. The clinicians' ranking of the models was strongly correlated with the judge's (Spearman ρ = 0.94), with the same top two and bottom two models, and 90.6% of their scores on individual conclusions were within one point of the judge's. We will include the additional clinician verification results, together with all repeated-run, multi-seed, and statistical significance analyses, in the revised manuscript.

Response to Q1: We thank the reviewer for asking about direct conclusion generation and independent specialist agents to clarify the actual benefit of discussion simulation. Holding the model and clinical inputs fixed, we evaluated 12 of the 14 model configurations using a conclusion-only prompt that bypasses discussion simulation. Simulating the discussion improves conclusion alignment for leading models including Grok 4.6 (+0.23), Gemma 4 31B (+0.14), and Gemini 3.7 Flash (+0.11), all exceeding run-to-run variation (up to 0.09), whereas other models perform comparably to direct generation (average gain of +0.026, within variation), indicating that stronger models are better able to convert multi-turn deliberation into more accurate decisions. Furthermore, the recorded discussions provide valuable training supervision, as reinforcement learning fine-tuning on these trajectories raises Board Simulation alignment by 17.7% (from 1.58 to 1.86).

To explore whether separate specialist agents grounded in the clinical evidence outperform single-model simulation while holding the base model (MedGemma 27B) and input evidence fixed, we implemented an independent multi-agent baseline. In this framework, a moderator selects three to six of ten specialist roles, each agent independently assesses the case summary and slides before responding to the other specialists, and the moderator synthesizes the final conclusion. This multi-agent setup achieves a score of 2.13, matching our single-model simulation (2.15) within run-to-run variation and demonstrating that single-model simulation captures multi-perspective deliberation just as effectively without the overhead of separate agent instances. We will add both baselines to the revised manuscript to provide a complete empirical comparison.

AuthorsOpenReview ↗

Response to Reviewer sfvw (3/3): Discussion quality and roles (Q2), reliability (Q3)

Response to Q2: We thank the reviewer for raising the importance of evaluating intermediate discussion quality and specialist role simulation beyond final conclusions. To assess the clinical correctness and structure of the generated turns while holding the clinical cases fixed, we scored each simulated discussion against the recorded tumor board transcript across six dimensions, namely fidelity to case facts, clinical reasoning, evidence coverage, specialist contribution, deliberation, and grounding of the conclusion. These metrics clearly separate model capabilities, with clinical reasoning spanning from 1.91 (MedReason-8B) to 4.95 (DeepSeek-V4-Flash), specialist contribution spanning from 1.81 (MedReason-8B) to 4.99 (Claude Opus 5), and unsupported case facts occurring in 0% of discussions for top models up to 58% for MedReason-8B. Furthermore, these evaluations are highly reliable across judges, as Claude Opus 5.5 produces nearly identical rankings across the first five dimensions (ρ = 0.89 to 0.99), and clinically meaningful, as reasoning, coverage, specialist contribution, and deliberation strongly track conclusion alignment under both judges (ρ = 0.70 to 0.81 and 0.71 to 0.79).

To examine how accurately models simulate distinct specialist roles and whether performance varies across specialties, we analyzed Specialist Turn responses across nine models, where each question is directed to a designated clinical role. Mean clinical equivalence scores and critical-error rates across addressed specialties are 2.85 (15.6% across 1,766 questions) for surgeons, 2.89 (17.4% across 207 questions) for molecular pathologists, 2.92 (16.0% across 1,347 questions) for medical oncologists, 3.00 (13.1% across 569 questions) for radiation oncologists, 3.06 (12.8% across 192 questions) for pathologists, 3.24 (10.2% across 407 questions) for radiologists, and 2.99 (13.0% across 307 questions) for other roles. Model rankings remain highly consistent across specialties (Spearman ρ = 0.93 to 1.00 against the overall ranking, with DeepSeek-V4-Pro ranking first and MedReason-8B ranking last in every major role), and the variation across roles (2.85 to 3.24) is substantially smaller than the variation across models (2.09 to 3.43), demonstrating that performance depends primarily on underlying model capability rather than the assigned specialty. We will include both analyses in the revised manuscript.

Response to Q3: We thank the reviewer for emphasizing evaluation reliability, clinician validation, and statistical significance testing. Regarding agreement with clinical experts, clinicians newly scored 180 Board Simulation conclusions from 30 test cases using the exact judge rubric while blinded to model identity. They ranked the six models almost identically to the judge (Spearman ρ = 0.94), and 90.6% of their scores were within one point of the judge's. To evaluate inference stability while holding the task and prompts fixed, we reran a subset of sampled configurations three to four times. On Board Simulation, the standard deviation is 0.009 to 0.039 and most adjacent gaps are 3 to 18 times larger, leaving only two near-tied configurations within run-to-run variation, while on Specialist Turn the standard deviation is 0.004 to 0.009 and the smallest adjacent gap is seven times the run-to-run range, with greedy configurations yielding identical outputs on every run. To confirm that training improvements are not seed-specific, we trained two additional reinforcement learning seeds that score 1.79 and 1.75 alongside the paper run of 1.86 (mean 1.80), with every seed outperforming supervised fine-tuning (1.67) and the base model (1.58).

To establish statistical significance across repeated generations and judge runs, we applied Welch's t-test over at least four complete runs per model with Benjamini-Hochberg correction, requiring the same directional sign under all three judges (Qwen3.8-27B, Claude Opus 5.5, and GPT-6.1 Sol). Under these criteria, 35 of 36 Specialist Turn comparisons (including every adjacent rank) and 78 of 91 Board Simulation comparisons are statistically separated, with Gemini 3.7 Flash leading every other configuration (t = 13.1, q less than 0.001), and we will explicitly report unseparated configurations as ties. Both training gains are likewise statistically significant, as supervised fine-tuning improves over the base model by 0.09 (t = 4.7, q = 0.004) and reinforcement learning improves over supervised fine-tuning by a further 0.21 (t = 10.6, q less than 0.001), holding the same direction under GPT-6.1 Sol. We will include all repeated runs, per-role tables, and discussion scores in the revised appendix to ensure complete transparency and reproducibility.

Reviewer_moweOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce OpenTumorBoard, a benchmark for evaluating LLMs on real-world multidisciplinary tumor board discussions. The dataset consists of 611 patient cases extracted from publicly available tumor board recordings, with specialist discussions and clinical conclusions. The authors propose two evaluation tasks: Specialist Turn, where an LLM responds to questions posed during discussions, and Board Simulation, where LLMs simulate the full discussion and arrive at a consensus. They evaluate several general-purpose and medical LLMs, and also demonstrate improvements with SFT and RL.

Critical strengths
  • I think the task itself is quite interesting. There have been several medical QA benchmarks, but evaluating LLMs on actual multidisciplinary tumor board discussions is a nice direction, especially with the complexity of cancer treatment decisions.

  • The dataset is fairly large, with 611 patient cases and over 19,000 discussion turns across different cancer types and specialist roles. Having actual discussion trajectories rather than synthetic clinical questions is a useful contribution.

  • I also like the idea of evaluating both individual specialist responses and full tumor board discussions, and the authors provide a fairly detailed pipeline for curating this data.

Critical weaknesses
  • My biggest concern is the heavy use of LLMs in constructing the benchmark itself. From Section 2.2, GPT-5.4 is used for almost everything, including case segmentation, role inference, evidence extraction, question extraction, rewriting, and extracting the final conclusions. These are quite important steps as they determine both the inputs and ground-truth answers. Although the authors do some human validation, I am not entirely convinced that this is enough. In Section 3.4, only 15 out of 611 cases are reviewed by clinical experts, and the conclusions about case factuality and consensus fidelity are based on these 15 cases. Appendix A also reports a separate audit of 15 recordings for the processing pipeline, but this does not necessarily establish the correctness of the extracted clinical information across the full dataset. Especially for complex medical discussions, I would expect more rigorous human verification. Also while the cases are real-world, and Table 1 shows the differences from existing benchmarks, some of them were not auto-generated QA.

  • I am quite skeptical of the evaluation protocol. Almost all the reported results rely on a single LLM judge, Qwen3.8-27B, assigning scores between 1 and 5. I don't think these point-based metrics are necessarily reliable enough for judging complex clinical recommendations, where even small differences can completely change the meaning of an answer. There is also no comparison of these scores with actual clinician judgments on model-generated outputs. The authors acknowledge this as future work, but I feel this is essential for a benchmark where the main contribution is evaluating clinical reasoning. How do we know that a model scoring 3.02 is actually better than one scoring 2.88? I would like to see human agreement, multiple judges, or at least some sensitivity analysis. How robust are the judges? In medical reasoning, often when the answer is open-ended, judges are used, but using them for the entire evaluation introduces bias. Similarly for unsupported claim-rates etc.

  • For TBs, the discussion also matters quite a bit. One can reach the right answer with completely wrong reasoning. From Appendix C, the judge only evaluates the generated conclusion rather than the entire discussion trajectory. However, the main motivation of the paper is to evaluate how LLMs can simulate multidisciplinary discussions, including interactions between specialists.

  • Can the authors compare to blind prediction? Basically predict the case without the discussion? I’m skeptical that the discussion trajectories don’t help a lot.

  • A big flaw in the entire TB setting is that treatment recommendations are never static. Guidelines keep changing, new drugs come to the market, experimental/off-label drugs are suggested in some cases. LLMs without any external source can have the wrong guidelines in memory, or not be upto-date. That makes everything very hard to evaluate. Sometimes treatments are derived from patient’s response to medication. This is why focusing on discussion trajectories is more important than final answer.

  • There are also some experimental confounders in the model comparisons. Some models receive slide images, while others only receive captions, and different reasoning configurations are used. I don't think these are entirely fair comparisons if the goal is to understand differences in clinical reasoning. Moreover, Figure 5 shows a correlation between discussion length and conclusion alignment, but this could simply be because stronger models generate longer responses.

Questions for authors

Mentioned along with weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer mowe (1/3): Benchmark verification (W1) and judge reliability (W2)

Response to W1: Thank you for raising the concern about how much of the benchmark has been verified. We extended the Appendix A audit from 15 recordings to all 219, and clinicians reviewed the case summaries and reference conclusions of 10 additional test cases beyond those in Section 3.4.

We used the original recordings as the reference for checking the extracted material. Across all recordings, we checked every speaker role, case boundary, and extracted slide frame, along with a random sample of 811 (5%) question and answer rewrites. The audit found that 97.8% of speaker roles were correct, 99.2% of case boundaries were correct or differed only slightly, and 99.5% of extracted frames were slides. Among the sampled rewrites, 0.7% changed the question’s meaning, and all preserved the answer’s meaning. In the clinical review of 447 pairs from the 15 cases in Section 3.4, three clinicians found that 94.4% of answers were fully supported by the recorded discussion and 92.6% were clinically correct. Following the reviewer's suggestion, we asked clinical experts to annotate 10 additional test cases, expanding our clinician-verified subset to 25 cases in total. In this newly annotated batch, all 10 reference conclusions (100%) faithfully captured the tumor board's final decisions, and 9 of 10 case summaries (90%) preserved the essential clinical information of the case. These outcomes closely mirror our findings on the initial 15 cases, confirming the stability and consistency of our dataset construction. Given that scaling to the entire dataset would require roughly 4,900 clinician-hours, this 25-case audit offers a practical and reliable validation of overall extraction quality, and we will incorporate these updated results into the revised manuscript. Clinicians have also scored model outputs on 30 test cases, including these 10, while blinded to model identity, as described in our response to W2.

We agree that several benchmarks in Table 1 use human-authored questions. OpenTumorBoard is the only one that uses actual QAs from tumor board meetings as its source. In each Specialist Turn, the question comes from a real clinician while discussing the patient’s case, and the reference answer comes from the specialist’s reply in that same meeting. GPT-5.4 only rewrites them into formal QA format. All 16,215 reference answers retain the clinician’s verbatim wording and can be checked against the source utterance. We will clarify in the introduction that each pair originates in clinical practice. MTBBench is the only existing benchmark for tumor board evaluation, however, 390 of 573 are generated by GPT-4o.

Response to W2: We appreciate the reviewer’s request to test whether small score gaps reliably distinguish models. To address these concerns, we have now incorporated blinded clinician validation, repeated generation runs with significance testing, and multi-judge reliability analyses. For the Specialist Turn comparison highlighted by the reviewer, Ministral 3 14B scored significantly higher than MedGemma 27B across at least four runs per model, with p = 0.0013. When all three judges scored the same outputs using the same rubric and references, each ranked Ministral higher.

On Specialist Turn, we ordered models by their mean scores under Qwen3.8-27B and compared each model with the one immediately below it. Every such pair differed significantly across repeated model runs, and all three judges agreed on which model scored higher. Across all possible model pairs, 97.2% on Specialist Turn and 85.7% on Board Simulation met both criteria. We tested differences with Welch’s t-tests and corrected for multiple comparisons using the Benjamini and Hochberg procedure. When Claude Opus 5.5 and GPT-6.1 Sol rescored all Board Simulation outputs and two questions per test case on Specialist Turn, their rankings correlated with the original judge’s at 0.983 to 1.00 on Specialist Turn and 0.956 to 0.977 on Board Simulation by Spearman’s measure. On individual Specialist Turn responses, quadratic-weighted kappa ranged from 0.865 to 0.909.

Across three repeat evaluations of the same outputs by Qwen3.8-27B at temperature 0.7, no model’s mean score shifted by more than 1.75% of the scoring range. The same runs reproduced 91.9% of unsupported-claim flags and 95.0% of critical-error flags. For both flag rates, Claude’s model rankings correlated with the original judge’s at Spearman ρ = 0.97.

To assess agreement with clinical judgment directly, clinicians newly scored 180 Board Simulation conclusions from 30 test cases, produced by six models, using the same rubric while blinded to model identity. The clinicians ranked the models almost identically to the judge (Spearman ρ = 0.94), with the same top two and bottom two models, and 90.6% of their scores on individual conclusions were within one point of the judge's.

AuthorsOpenReview ↗

Response to Reviewer mowe (2/3): Discussion evaluation (W3) and blind prediction (W4)

Response to W3: We thank the reviewer for emphasizing the importance of evaluating how specialists reason and interact throughout a tumor board discussion. Our Specialist Turn task already evaluates key intermediate points in real discussions by asking the model to respond as the addressed specialist at a specific point in the meeting. Following the reviewer’s suggestion, we extended Board Simulation to evaluate the full discussion trajectory. We compare each simulated discussion with the recorded discussion on six dimensions: fidelity to case facts, clinical reasoning, coverage, specialist contribution, deliberation, and grounding of the conclusion. The rubric gives credit for the substance of the discussion regardless of its length.

The new evaluation corroborates the original conclusion-based comparison across models. Scores for clinical reasoning, coverage, specialist contribution, and deliberation each correlate strongly with final conclusion alignment (Spearman ρ = 0.70–0.81). Models that better reproduce how specialists contribute and deliberate also tend to reach conclusions that more closely match those of the recorded boards. We will report all six dimension scores for every model in the revision.

Response to W4: We thank the reviewer for suggesting the blind-prediction comparison. Our experiments show that discussion trajectories help in two ways. Generating a discussion can improve a model’s conclusion at inference time, and recorded discussions provide valuable supervision during training.

At inference time, we ran the suggested comparison for 12 of the 14 configurations. Each model either predicted the conclusion directly or first worked through the case in an explicit, turn-by-turn discussion among specialists. Generating the discussion improved conclusion alignment for Gemini 3.7 Flash (+0.11 points), Grok 4.6 (+0.23), and Gemma 4 31B (+0.14), with gains exceeding run-to-run variation. For the other models, the two approaches performed comparably, with a mean difference of only 0.026 points within run-to-run variation. These results show that several of our strongest models benefit from working through the discussion before reaching a conclusion, and such discussion capability varies across models.

During training, the recorded trajectories show how specialists interpret evidence, weigh options, and arrive at a conclusion. Using these trajectories to train Qwen2.5-VL-3B-Instruct raised Board Simulation alignment from 1.58 to 1.67 with SFT (+5.7%) and to 1.86 with RL (+17.7%). Across three RL seeds, the mean score was 1.80 (+14% over the base model), and all three runs outperformed SFT.

AuthorsOpenReview ↗

Response to Reviewer mowe (3/3): Evolving guidelines (W5) and comparison confounders (W6)

Response to W5: We fully agree with the reviewer’s insightful observation that tumor board decision-making is dynamic, driven by evolving guidelines, emerging therapies, and individual treatment responses, which makes trajectory evaluation essential.

In our benchmark setting, outdated parametric knowledge matters less because models do not decide from memory alone. Each test case provides the clinical evidence and slides presented at the meeting, which frequently discuss newly approved drugs, ongoing clinical trials, or off-label options under consideration. Furthermore, 151 of the 184 test cases (82.1%) include longitudinal histories detailing prior lines of therapy and the patient’s specific responses and outcomes as part of the input summary. Thus, models are evaluated on their ability to reason over the provided patient trajectory and contemporary evidence rather than reciting static guidelines from memory.

We also expanded our evaluation framework to score the discussion trajectories directly, with full results detailed in our response to W3. Moreover, because every case in our release includes its exact upload date, each reference standard is naturally anchored to the standard of care at that specific point in time. This temporal metadata allows future work to stratify evaluations by date or supply the clinical guidelines in force at the time of the meeting. Finally, we have updated the limitations section to note that a model’s internal guideline knowledge may lag behind evolving clinical standards and that each reference reflects the standard of care at the time of the meeting.

Response to W6: The reviewer raises a thoughtful point regarding the consistency of input modalities, reasoning settings, and output lengths across model comparisons. Because real-world tumor boards review presentation slides alongside clinical discussion, multimodal perception serves as an inherent dimension of the evaluation, and text-only models receive brief slide captions simply because their architectures cannot process images, which naturally highlights the value of multimodal clinical benchmarking. Similarly, to keep reasoning settings as consistent as possible across diverse model APIs, we treated the reasoning configuration as a standardized hyperparameter by running every reasoning-capable model with reasoning enabled at its highest or default active setting, specifically xhigh effort for Gemini 3.7 Flash, Grok 4.6, Qwen3.8-Max, and GPT-5.6 Sol, thinking enabled for DeepSeek-V4, and adaptive thinking for Claude Opus 5. To further verify that our comparative findings are robust to these hyperparameter choices while holding the task fixed, we also tested four models without reasoning, where taking each model's stronger run leaves the rankings nearly unchanged with Kendall's τ of 0.99 and 0.94. Varying effort levels across the top three models likewise preserves their relative order, as Gemini 3.7 Flash remains first at every level with 2.78 to 2.81 on Board Simulation (2.79 in the paper) compared with 2.54 to 2.65 for Grok 4.6 (2.58) and 2.51 to 2.68 for Claude Opus 5 (2.53).

We also appreciate the reviewer's insightful observation regarding Figure 5 and agree that the plot captures an observational correlation rather than a naive causal claim that verbosity alone improves accuracy. In fact, the reviewer's point that stronger models generate longer responses reflects the exact mechanism underlying complex clinical deliberation, much like how capable reasoning models leverage extended chain-of-thought to solve difficult problems whereas weaker models cannot translate extra tokens into valid reasoning. Stronger models achieve better conclusions precisely because they can sustain longer, substantive multi-specialist discussions that weigh competing evidence rather than merely producing verbose text. Our controlled experiments directly confirm this mechanism. When holding the base model fixed, engaging in simulated board discussion improves conclusion alignment over direct answering by +0.23 for Grok 4.6, +0.14 for Gemma 4 31B, and +0.11 for Gemini 3.7 Flash, all exceeding run-to-run variation (W4). Furthermore, when scoring discussion trajectories directly without rewarding length (W3), substantive reasoning, evidence coverage, specialist contribution, and deliberation closely track conclusion alignment across models with Spearman ρ of 0.70 to 0.81 and 0.71 to 0.79 under Claude Opus 5.5, so we will clarify the correlational nature of Figure 5 and pair it with these trajectory-quality findings to provide a complete view of how capable models utilize extended deliberation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.