- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OpenTumorBoard benchmarks multidisciplinary tumor board decision-making using 611 multimodal patient cases from public YouTube recordings. Its two tasks, SPECIALIST TURN and BOARD SIMULATION, evaluate specialist responses and consensus formation. Evaluations of 14 LLMs reveal substantial performance gaps, with supervised finetuning and reinforcement learning yielding modest gains.
Critical strengths
-
OpenTumorBoard captures real tumor board discussions across ten specialist roles and combines multimodal clinical evidence with longitudinal patient histories. Its two tasks evaluate specialist responses and board-level consensus, closely reflecting real clinical workflows.
-
The paper presents a clear automated pipeline for transcription, role identification, slide extraction, and temporal alignment. Expert audits and component-level checks assess data quality and pipeline reliability.
Critical weaknesses
-
The curation pipeline relies heavily on LLMs for role identification, case segmentation, and question and conclusion extraction. Errors in these steps could propagate through the dataset, and the human audits cover only a limited sample.
-
The RL experiments use LLM judges to provide rewards and measure improvements. This setup may favor responses that satisfy the judge without improving clinical quality. More human evaluation would help establish whether the reported gains reflect meaningful improvements.
-
The main evaluation relies on a single judge model, Qwen3.8-27B, which may influence model rankings. The clinician evaluation covers only 15 cases, and the paper does not report the correlation between clinician ratings and judge scores.
-
The cases come from publicly available YouTube recordings and may differ systematically from routine clinical cases. The paper provides limited evidence that the benchmark represents different institutions, regions, and tumor board formats.
Questions for authors
-
What slide content will you release: captions and source URLs with timestamps, or extracted slide images as well? If images are not redistributed, how can others reproduce the multimodal benchmark at scale?
-
How did you quantitatively validate the removal of information leakage? Could you report key statistics in the main text, such as the proportion of turns or frames removed, and checks confirming that the inputs do not reveal the final conclusions?
-
Did you evaluate the systems using multiple independent LLM judges? Could you report rank correlations and the stability of system rankings across judges, or add this analysis if it has not been conducted?
-
Could you evaluate the RL improvements using a different judge family and a small clinician-rated subset? This would help assess whether the gains reflect better clinical responses or overfitting to the reward judge.
-
How representative are the YouTube-sourced tumor boards in terms of geography, language, academic versus community settings, and case acuity? Could you quantify the concentration of cases by institution and cancer type, and discuss plans to include data from additional institutions?
4 comments
Response to Reviewer sdAB (2/3): Judge robustness (W3) and representativeness (W4)
Response to W3: We thank the reviewer for the suggestion to examine how much the rankings depend on a single judge and how judge scores relate to clinician ratings. We rescored model outputs with two judges from other model families, Claude Opus 5.5 and GPT-6.1 Sol, using the same rubric and references as the original judge.
On Board Simulation, both judges rescored all outputs, and their model rankings correlated with the original judge's at Spearman ρ = 0.977 (Claude Opus 5.5) and 0.956 (GPT-6.1 Sol). Gemini 3.7 Flash ranked first under all three judges. On Specialist Turn, where the two judges rescored every model's responses to two questions per test case, the rankings correlated at ρ = 0.983 to 1.00, quadratic-weighted kappa between judges on individual responses ranged from 0.865 to 0.909, and all three judges agreed on the order of every pair of adjacent models.
To measure the correlation between clinician ratings and judge scores, clinicians newly scored 180 Board Simulation conclusions from 30 test cases with the judge's rubric while blinded to model identity, and the expanded clinician review described in our response to W1 goes beyond the original 15 cases. The clinicians' ranking of the models correlated with the judge's at Spearman ρ = 0.94, and 90.6% of their scores on individual conclusions were within one point of the judge's.
Response to W4: We thank the reviewer for asking how representative the recordings are. To quantify this along the dimensions the reviewer lists, we characterized all 611 cases by geography, language, hosting setting, acuity, channel, and cancer site, using YouTube channel metadata and our case annotations.
By geography, 34.0% of cases come from the United States, 16.9% from international multi-country boards, and 16.4% from other countries, while 32.7% could not be determined from the metadata. Speech is in English in 215 of the 219 recordings. By hosting setting, 38.1% of cases come from professional societies and conferences, 24.5% from patient advocacy organizations, 17.7% from medical education media, 2.9% from academic medical centers, and 1.0% from community oncology, with 15.7% other or undetermined. By acuity, 38.0% of cases are metastatic, 26.8% localized, 18.7% locally advanced, and 12.8% recurrent. The cases are spread across many sources: the largest of the 62 channels holds 17.0% of cases (Herfindahl index 0.076, equivalent to about 13 equally sized channels), and the largest of the 17 cancer sites, thyroid and parathyroid, accounts for 20.6%.
These boards are broadcast for education and likely favor complex cases, which is the setting where multidisciplinary review matters most, and we will state this in the limitations. Routine tumor boards are rarely shared because of patient privacy, so public recordings are currently the main source that can be openly released. To include more institutions, our released pipeline allows any institution to add its own recordings, and we welcome such contributions.
Response to Reviewer sdAB (3/3): Questions Q1–Q5
Response to Q1: We will release each slide's caption, source YouTube URL, and timestamp, together with the slide-to-case index, the data splits, and the full curation pipeline. With these, others can regenerate the slide images from the public recordings and reproduce the multimodal benchmark at scale.
Response to Q2: We validated leakage removal by measuring how much of each input was removed and by checking whether the edited inputs still reveal the decision. De-leaking edits only the model inputs and never the references. Before de-leaking, a classifier confirmed post-decision information in 58 of the 184 test cases (31.5%). De-leaking changed the Board Simulation inputs for 88 test cases (47.8%), removing 9.5% of input text tokens and 94 of 1,435 slides (6.6%), and removed 149 of 4,993 Specialist Turn test questions (3.0%). To check whether the text inputs still reveal the decision, two LLMs not used in curation, DeepSeek-V4-Pro and Gemini 3.7 Flash, judged whether each case's inputs state or imply the final conclusion. They flagged 80 and 79 cases before de-leaking and only 16 and 13 after. Excluding the 20 cases flagged by either model after de-leaking moves no configuration by more than one rank (ρ = 0.991). We will report these statistics in Section 2.2.
Response to Q3: Yes. In addition to Qwen3.8-27B, we rescored model outputs with Claude Opus 5.5 and GPT-6.1 Sol. The rank correlations and the stability of the rankings across the three judges are reported in our response to W3.
Response to Q4: Yes. As described in our response to W2, the ordering of the base, SFT, and RL models is unchanged under GPT-6.1 Sol, a judge from a different model family, and RL retrained with Gemma 4 31B as the reward judge also remains above SFT and the base model. Clinicians blinded to model identity ranked the three models in the same order, scoring the base, SFT, and RL models 1.93, 2.33, and 2.70 on 30 test cases.
Response to Q5: Please see our response to W4, which reports the distribution of cases by geography, language, hosting setting, and acuity, the concentration of cases by channel and cancer site, and our plan for including data from additional institutions.
Thank you for the detailed responses and additional experiments. These have satisfactorily addressed my main concerns. I look forward to seeing the additional results and clarified limitations incorporated into the revised manuscript.
Response to Reviewer sdAB (1/3): Curation quality (W1) and RL robustness (W2)
Response to W1: Thank you for raising the concern that errors in the LLM-based curation steps could propagate through the dataset. We extended the Appendix A audit from 15 recordings to all 219, and clinicians reviewed the case summaries and reference conclusions of 10 additional test cases beyond those in Section 3.4.
We used the original recordings as the reference for checking each curation step. Across all recordings, we checked every speaker role, case boundary, and extracted slide frame, along with a random sample of 811 (5%) question and answer rewrites. The audit found that 97.8% of speaker roles were correct, 99.2% of case boundaries were correct or differed only slightly, and 99.5% of extracted frames were slides. Among the sampled rewrites, 0.7% changed the question's meaning, and all preserved the answer's meaning. Because errors are rare at every step, few can propagate into the model inputs and references. In addition, all 16,215 Specialist Turn reference answers retain the clinician's verbatim wording and can be checked against the source utterance. In the clinical review of 447 pairs from the 15 cases in Section 3.4, three clinicians found that 94.4% of answers were fully supported by the recorded discussion and 92.6% were clinically correct. Following the reviewer's suggestion, we asked clinical experts to annotate 10 additional test cases, expanding our clinician-verified subset to 25 cases in total. In this newly annotated batch, all 10 reference conclusions (100%) faithfully captured the tumor board's final decisions, and 9 of 10 case summaries (90%) preserved the essential clinical information of the case. These outcomes closely mirror our findings on the initial 15 cases, confirming the stability and consistency of our dataset construction. Given that scaling to the entire dataset would require roughly 4,900 clinician-hours, this 25-case audit offers a practical and reliable validation of overall extraction quality, and we will incorporate these updated results into the revised manuscript. Clinicians have also scored model outputs on 30 test cases, including these 10, while blinded to model identity, as described in our response to W2.
Response to W2: We thank the reviewer for raising the question of whether the RL gains extend beyond the reward judge. We tested this with an evaluation judge from a different model family, with a different reward judge, and with blinded clinician scoring of the RL outputs.
We first rescored the base, SFT, and RL models on Board Simulation with GPT-6.1 Sol, which belongs to a different model family from both the RL reward judge (Qwen3.6-27B) and the paper's evaluation judge (Qwen3.8-27B). The ordering was unchanged, with the base model at 1.64, SFT at 1.74, and RL at 1.82. We then retrained RL with Gemma 4 31B as the reward judge instead of Qwen3.6-27B. The final checkpoint scored 1.87 under the paper's judge and 1.79 under GPT-6.1 Sol, compared with 1.86 and 1.82 for the original RL model, and it remained above SFT and the base model under both judges.
To assess whether these gains reflect better clinical responses, clinicians newly scored the base, SFT, and RL conclusions on 30 test cases with the conclusion alignment rubric, blinded to model identity and mixed with outputs from three other models. The clinicians gave the base, SFT, and RL models 1.93, 2.33, and 2.70, the same ordering as the judges, and scored RL above the base model in 14 cases and below it in only 5. They also scored RL (2.70) close to DeepSeek-V4-Pro (2.73) and Gemini 3.7 Flash (2.80) and above MedGemma 27B (2.60).