acceptodds
Under review as a conference paper at ICLR 2027

DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis

Abstract

Diagnosing a patient from video requires more than recognizing the sign: a vision–language model must turn what it sees into hypotheses, questions and tests. DynamicDx evaluates each step in 71 neurological consultations across 11 sign categories, linking authentic patient videos to confirmed diagnoses and fixed charts built from the same case reports, so that every model queries the same evidence. Across five such models, video improves accuracy by 9.9–22.5 percentage points over blind input, but neither recognition alone nor temporal order explains the gain: the cause is usually missing from the model's video-only differential diagnosis even when the sign is recognized, and shuffling the frames produces no reliable accuracy loss. Instead, a trajectory replay traces most of the gain to the investigation results the video prompts. Evidence acquisition is the bottleneck: supplying the decisive investigations raises accuracy to 73.2–93.0%. Two interventions act on it. A post-trained 4B video describer improves sign descriptions, especially from a short, densely sampled segment, and source-clean literature retrieval expands initial hypotheses; both bring the tests a model orders closer to those the treating clinicians documented and, through them, raise accuracy. For video-based diagnosis, seeing better helps when it leads to asking better.

Primary area:
datasets and benchmarks
Keywords:
Multimodal large language models, Clinical video understanding, Diagnostic reasoning, Evidence acquisition, Interactive clinical benchmarks, Neurological diagnosis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Fhy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DynamicDx is a process-level benchmark for video-based diagnosis: 71 real patient videos across 11 neurological sign categories, each paired with the confirmed diagnosis and a fixed chart built from the source case report. Stage 1 tests reactive perception (describe the sign, propose a differential); Stage 2 tests proactive evidence acquisition (history questions, investigation orders, final diagnosis) against the fixed chart. Five VLMs are compared under controlled conditions (blind, single frame, shuffled, own words, clinician reference, oracle evidence), plus two interventions: a post-trained temporal-window describer and source-clean literature retrieval. The headline finding is that video helps mainly through the investigations it prompts, not through recognition or temporal order.

Critical strengths

DynamicDx links real patient videos to confirmed diagnoses and a fixed chart from the same case report, and it scores recognition, history taking, test ordering, and final diagnosis separately. That makes it possible to see where a model fails, not just whether the final answer is right.

The second strength is that the central claim rests on more than a single accuracy number. The paper shows that video mainly helps through the tests it prompts, and the trajectory replay and oracle evidence results point the same way.

Critical weaknesses

The evidence base is small and partly synthetic. The benchmark has only 71 cases, with one category containing a single clip and several categories below 10 cases. More importantly, 5,972 of 6,398 chart entries are derived rather than reported, and the paper itself shows that removing derived values lowers accuracy by about ten points. This makes absolute accuracy and some contrasts difficult to interpret as measures of real diagnostic behavior.

The main empirical pattern is mixed, but the narrative is stronger than the evidence. Multi-turn consultation improves three models and degrades two; ordered video exceeds shuffled frames by only 5.4 points with an interval including zero (-0.3, +11.3); own words change accuracy by -2.8 to +1.4 points; and retrieval gains remain only for MiMo-v2.5 and MiniMax-M3. The conclusion that “seeing better helps when it leads to asking better” is plausible but not established at the claimed strength.

Reproducibility is limited. The pipeline depends on several proprietary or rapidly changing models, including GPT-5.6-luna, DeepSeek-V4.1-Flash, and Qwen3.8-flash, and no code or model release is described. The source videos are not redistributed for licensing reasons, which further limits independent reproduction.

Questions for authors

Which conclusions survive a correction for multiple comparisons, given that several key intervals include zero?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Fhy4: W1

W1 (summarised). The evidence base is small and partly synthetic: 71 cases, one category with a single clip and several below ten; 5,972 of 6,398 chart entries are derived, and removing derived values changes accuracy by about ten points. Absolute accuracy and some contrasts are therefore hard to interpret.

Thank you for raising this important point. We agree that 71 cases is a small sample and that most chart entries are derived rather than reported, and the revision states both limits more clearly. We would also like to explain why we believe the main conclusion still holds.

  • Benchmark size limitation due to difficulty of data collection. Each case must combine an authentic patient video, a confirmed diagnosis and a documented work-up, all from the same open-access report. Building one means reading the full paper and summarising its history, examination, investigations and diagnosis by hand. A case report usually describes a single patient, so one paper yields one case, and most case reports do not include a video at all. In total, 66 reports met these requirements and gave 71 cases. Related benchmarks of interactive multimodal diagnosis are of the same order: For video-based diagnosis paper, Multimodal AMIE (Google DeepMind and Google Research; Nature Medicine) uses 105 scenarios (Saab et al., 2026) and the AI co-clinician study (Google DeepMind and Google Research; arXiv preprint) uses 20 (Shah et al., 2026), and both rely on patient actors rather than the videos of real patients that we use (Table 1).
  • Small categories. The category sizes follow what the literature provides. Facial palsy has a single clip because only one of the facial-palsy reports we screened included a video that met these requirements. We show per-category results for completeness only and draw no conclusions at the category level.
  • The key evidence is mostly real. Of the 301 decisive entries (the test results that settle a diagnosis), 256 are taken directly from the source reports.
  • The conclusion holds regardless of derived values. To show that the derived values do not change our conclusions, we ran two experiments that remove them while keeping each consultation's questions, answers and orders fixed: (i) replacing the 1,086 derived values that state a specific finding with a neutral one (Appendix C, Table 14), and (ii) withholding every derived entry, so that the chart returns only what the source article reports (Appendix C.4; see our reply to 6aM3-Q1). We summarise the results for the three conditions in the table below.
ChartBlindVideoOracle evidenceVideo − blind
Original (Table 2)13.830.182.0+16.3
(i) Derived findings neutralised9.319.480.0+10.1
(ii) Reported entries only7.622.377.5+14.6

Accuracy (%), mean of five systems, 71 cases. Row (i) re-runs the diagnosis turn; the same re-run on the original chart gives 14.4, 29.3 and 81.4 (+14.9). Row (ii) applies the change measured in the replay of 6aM3-Q1 to the accuracies of Table 2.

Removing the derived values lowers absolute accuracy in every condition, but the ordering does not change: accuracy with video remains higher than with blind input, and accuracy with oracle evidence remains substantially higher than with video.

In the revision, we made three changes:

  • Conclusion: the limitations now quantify how much derived values contribute to accuracy and to the benefit of video (see our reply to 6aM3-Q1).
  • Section 4.2: the text now adds that “withholding all derived entries narrows it from 16.3 to 14.6”, where “it” is the benefit of video over blind input.
  • New Appendix C.4 and C.5: C.4 reports accuracy when the chart returns reported entries only, and C.5 repeats the main comparisons on the cases with mostly real charts and on those with more answerable records.
AuthorsOpenReview ↗

Response to Reviewer Fhy4: W2

W2 (summarised). The narrative is stronger than the evidence: multi-turn helps three models and hurts two; ordered minus shuffled is 5.4 points with an interval including zero ([−0.3, +11.3]); own words change accuracy by −2.8 to +1.4; retrieval gains hold only for MiMo-v2.5 and MiniMax-M3. “Seeing better helps when it leads to asking better” is plausible but not established at the claimed strength.

Thank you; we agree that parts of the narrative were stated more strongly than the evidence supports. We have rewritten each of the four points you list, and the closing sentence. We also explain which results the central claim rests on.

What we changed (old → new).

  • Frame order: “shuffling the frames produces no reliable accuracy loss” → “frame order matters for some models but not for others”, with the loss of each model given in Section 4.2 (see our reply to 6aM3-W2).
  • Multi-turn consultation: Section 4.2 no longer says that the conditions “perform alike”; it says that their “effects differ between models”. Multi-turn consultation helps GPT-5.6-luna (+5.6 points), MiniMax-M3 (+8.4) and Qwen3.8-flash (+1.4), and hurts Gemma-4-31B (−7.0) and MiMo-v2.5 (−2.8).
  • Own words: the −2.8 to +1.4 point range is now called “no detectable difference” rather than equivalence between the two inputs.
  • Retrieval: Section 4.4 now states that “the MiMo-v2.5 gain survives Holm correction” (Holm-adjusted p = 0.016; for MiniMax-M3, p = 0.16), and that the retrieved content itself improves accuracy only for MiMo-v2.5 and test selection for MiMo-v2.5 and Qwen3.8-flash (see our reply to 6aM3-W4).
  • Closing sentence: “seeing better helps when it leads to asking better” → “what a model sees helps mainly through the tests it prompts”.

What the central claim rests on. The new closing sentence follows from two results, not from the overall pattern. Both have confidence intervals that exclude zero, and both survive a Holm–Bonferroni correction over all 27 effects the paper reports, listed in our reply to Fhy4-Q1 (Holm p ≤ 0.003).

  • Replay: the investigations from the video consultation add 12.1 points (95% CI 6.5 to 17.7), while its history adds only 0.3 (Table 11).
  • Oracle evidence: supplying the decisive investigations raises accuracy by 39.4–59.2 points over video, in each of the five systems.
AuthorsOpenReview ↗

Response to Reviewer Fhy4: W3

W3 (summarised). Reproducibility is limited: the pipeline depends on proprietary or fast-changing models (GPT-5.6-luna, DeepSeek-V4.1-Flash, Qwen3.8-flash), no code or model release is described, and the source videos are not redistributed.

Thank you for raising reproducibility; we may not have made the release visible enough. The data, prompts and evaluation code are available at https://anonymous.4open.science/r/dynamicDx-53C8 (Reproducibility statement). The README gives the exact command for every condition, and every prompt is reproduced in full in Appendix H. In the revision, the conclusion also points to this release.

  • Videos. 18 of the 66 source articles use the CC BY-NC-ND licence, which does not allow edited copies to be shared, so we cannot distribute the trimmed clips. Instead, the repository includes a script that downloads each article's original videos from Europe PMC and cuts every clip from them. data/clips.json records each clip's source file and start and end time, and, for the eight clips taken from split-screen or two-view recordings, the crop box; re-cutting all 71 clips this way reproduces the benchmark clips up to re-encoding differences.
  • Open-weight models. Of the seven models in the pipeline, six have openly released weights. Among the five evaluated systems these are Gemma-4-31B, MiniMax-M3 and MiMo-v2.5, and Qwen3.8-flash, whose weights are released as Qwen3.8-Flash-Next. The other two are DeepSeek-V4.1-Flash, which grades outcomes and maps questions and orders, and Qwen3.5-4B, the describer we post-train. We ran the evaluated systems through hosted endpoints, and the evaluation code accepts any model served through the same API, so the benchmark can be re-run on locally served weights.
  • GPT-5.6-luna is the only model without open weights. Evaluating a hosted frontier model is common in this area: CRAFT-MD (Harvard and Stanford; Nature Medicine; Johri et al., 2025) and AgentClinic (Johns Hopkins and Stanford; npj Digital Medicine; Schmidgall et al., 2026) evaluate hosted models such as GPT-4, and AMIE (Google; Nature; Tu et al., 2025; McDuff et al., 2025) and Multimodal AMIE (Google; Nature Medicine; Saab et al., 2026) are themselves systems whose weights are not released.
  • Saved outputs. Every call uses a fixed provider endpoint and temperature 0. Because hosted models change over time, the repository also includes the model outputs behind Table 2 (runs/): all 2,838 consultations of the eight conditions for the five systems, each with its questions, answers, orders, released chart entries and final diagnosis. The released grader can re-grade them without calling any model.
Reviewer_wZXhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a benchmark for the automatic diagnosis of neurological diseases, including video-based analysis. They study the relevance of different information sources (e.g. the video, the temporal ordering within the video, the querying of specific tests, literature retrieval) via controlled experiments. They contribute the benchmark itself and studied several models using the benchmark.

Critical strengths

1. Thorough Analysis with Controlled Conditions

The authors included many analyses in their benchmark, including studying the model performance under controlled conditions (i.e., controlling what information is given to the model). These controlled conditions enable the identification of potential issues with the model and enable studying the benchmarked models beyond simple performance comparisons. Thereby, they do not only contribute the benchmark itself but also several interesting findings.

2. Well-described dataset characteristics including cohort details

The authors provide many details about their dataset including a detailed source attribution and fine-grained analysis of cohort characterstics. This transparency enables the identification of potential biases or imbalance within the dataset, which can increase the trust in their benchmark.

Critical weaknesses

1. Clinician Baseline May be Misrepresented

The clinician baseline presented in Sec. 4.1 (Fig. 5) is presented as scoring extremely low (9.9% in phenomenology and 2.8% in differential). However, this baseline seems to be unreliable and may misrepresent a clinician’s performance due to the following reasons:

  • results are only based on a single neurologist’s judgement and the authors do not provide details about the neurologists experience level (e.g. board certification, seniority)
  • the neurologist rates ~30% of clips as unidentifiable. However, this could also mean that the neurologist is more cautious when rating clips while the models may use guesses/estimates when judging clips where diagnosis is complex. Note that the best models rate about 30% of phenomenology and about 15% of differentials correctly. Therefore, the experiments do not clearly prove that models are better at rating the clips.

Overall, this does not invalidate the central findings and the benchmark remains valuable, but authors need to be more transparent about this to assure findings are not misrepresented.

2. Automatic Labeling

Some annotations within the dataset are generated or adapted via an LLM (DeepSeek-V4.1-Flash), including:

  • Derived chart values given the confirmed diagnosis
  • Mapping of free-text questions and orders to the chart
  • Outcome grading
  • Normalization of the predicted phenomenology

While most of these are evaluated by human annotators, a risk remains of biases or hallucinations introduced via this automatic labeling. These risks should be clearly discussed.

Questions for authors
  • How is the question mapping (via DeepSeek) validated?
  • Are we sure that semi-synthetic investigation results (E) cannot be distinguished from real ones and thereby leak information about the case to the consultation model?
6 comments
AuthorsOpenReview ↗

Response to Reviewer wZXh: W1

W1 (summarised). The clinician baseline (Sec. 4.1, Fig. 5; 9.9% phenomenology, 2.8% differential) may be misrepresented: it rests on one neurologist whose experience is not reported, and judging about 30% of clips unidentifiable may reflect caution where models guess. The comparison does not show that models are better; the paper should be more transparent about this.

Thank you for this helpful comment. We agree that the clinician reference needs more context. The clinician is a neurocritical-care physician, not a neurologist, and the revision corrects this throughout. Three points help put the clinician's numbers in context:

  • List length. The clinician gave 2.3 diagnoses per clip, while the models gave 5.1–6.4. Coverage counts a correct cause anywhere in the list, which favours the models.
  • Caution rather than error. As you suggest, the clinician marked 22 clips as not identifiable instead of guessing. To show that these abstentions reflect caution rather than clips the models read better, we used the judgements as an independent label of how readable each clip is and compared the models' recognition of the sign on the 49 readable clips with the other 22 (Appendix B, Table 8). The table below summarises the results for each system.
SystemReadable (49 clips)Not readable (22 clips)Difference [95% CI]
GPT-5.6-luna30.622.7+7.9 [−14.4, +29.2]
Gemma-4-31B14.322.7−8.4 [−28.7, +11.3]
MiMo-v2.526.522.7+3.8 [−18.5, +24.0]
MiniMax-M310.218.2−8.0 [−27.7, +9.7]
Qwen3.8-flash26.527.3−0.7 [−23.5, +21.1]

Clips in which the model recognises the sign (%), 32 frames; readability judged by the clinician.

Even on the 49 clips the clinician found readable, the models recognise the sign in only 10.2–30.6% of cases, and no model differs reliably between the two groups. We conclude that the clinician's abstentions do not mark clips on which the models see more than the clinician did.

  • Full consultation. To show how the clinician compares once history and test results are available, the same clinician also completed the video condition of the full consultation (Table 2). The table below summarises the results.
Accuracy (%) [95% CI]QuestionsTests orderedDecisive tests (%)
Clinician35.2 [23.9, 46.5]4.63.118.5
GPT-5.6-luna46.5 [35.2, 57.7]16.28.528.9
Gemma-4-31B33.8 [22.5, 45.1]7.93.520.3
MiMo-v2.522.5 [12.7, 32.4]8.26.216.3
MiniMax-M325.4 [15.5, 35.2]11.37.719.3
Qwen3.8-flash22.5 [12.7, 32.4]6.35.316.3

Video condition, 71 cases. Questions and tests ordered are means per case.

The clinician reached 35.2% accuracy, second only to GPT-5.6-luna, while asking fewer questions and ordering fewer tests than any model. We conclude that, with history and test results available, the clinician is competitive with the strongest models while asking and ordering less than any of them.

In the revision, we made four changes:

  • Appendix B now reports the clinician's background: “six years of clinical practice in an emergency intensive care unit, subspecialties in neurocritical care and neurosurgery, and a professional Master of Medicine in neurosurgery”.
  • Sections 4.1 and 4.2, Figure 5, Appendix B and Table 8: “neurologist” → “neurocritical-care physician” or “clinician” throughout.
  • Section 4.1: the sentence “most model configurations exceed both rates” becomes “the two are not directly comparable: the models list more diagnoses and always answer, whereas the clinician declined when a clip could not be judged from video alone”.
  • Conclusion: limitations add “the clinician reference comes from one physician”.
AuthorsOpenReview ↗

Response to Reviewer wZXh: W2

W2 (summarised). Several annotations are generated or adapted by an LLM (DeepSeek-V4.1-Flash): derived chart values, mapping of free-text questions and orders to the chart, outcome grading, and phenomenology normalisation. Even with human checks, the risk of bias or hallucination should be discussed.

Thank you; we agree, and the revised conclusion now discusses it explicitly. Below we summarise how the LLM steps are constrained and checked.

How the LLM steps are constrained.

  • All derived chart values were fixed before any model was evaluated: 2,473 were written by annotators for each category's test menu, and 3,499 were filled in by DeepSeek-V4.1-Flash for a generic panel, given the confirmed diagnosis and the reported results.
  • During a consultation, the LLM only matches a question or order to entries already in the record and never writes a new finding, so every model gets the same record.

How they were checked against humans (Appendix J).

  • Grading: agreement with an independent human reviewer is 89.6% for sign recognition and 96.9% for aetiology (96 Stage 1 responses), and 91.5% for final diagnosis (71 GPT-5.6-luna video consultations; weighted κ = 0.947; disagreements one grade apart).
  • Test matching: 95 of 100 randomly sampled orders were matched correctly (3 returned extra results and 2 missed a result).
  • Retrieval: human re-grading reproduces the retrieval gains.

Remaining risks, which the revised conclusion now states.

  • Derived values may be more typical than real ones. The 3,499 LLM-filled values describe what the generator expects for the confirmed diagnosis, so they can be closer to a textbook case than a real result is. To show how much the results depend on them, we re-ran the diagnosis with the derived findings neutralised and with every derived entry withheld (Appendix C, Table 14; Appendix C.4; reply to 6aM3-Q1). The table below summarises accuracy under each chart.
ChartBlindVideoOracle evidenceVideo − blind
Original (Table 2)13.830.182.0+16.3
Derived findings neutralised9.319.480.0+10.1
Reported entries only7.622.377.5+14.6

Accuracy (%), mean of five systems, 71 cases. The neutralised row re-runs the diagnosis turn; the same re-run on the original chart gives 14.4, 29.3 and 81.4 (+14.9).

The conclusion that video improves accuracy over blind input, and that oracle evidence improves it far more, does not change, although derived values add about 5–10 points of absolute accuracy with and without video.

  • One model performs several steps. DeepSeek-V4.1-Flash derives values, maps questions and orders to the chart, and grades outcomes. A bias shared by these steps would not appear as disagreement between them; only the human checks above, which cover samples of the grading and the matching, can detect it.
  • History-question mapping. To show how reliable this step is, we audited 200 of the 8,804 mapped questions, 100 sampled from those the mapper answered yes and 100 from its other answers, and checked each against the case's symptom table (Appendix J.4; reply to wZXh-Q1). The table below summarises the audit.
Mapper answerShare of all questionsSampledWrongError rate (weighted)
yes8.4%10042.2%
Other answers91.6%10000.0%
All answers100%20040.2%

Four of the 100 sampled yes answers were wrong and none of the others, which corresponds to an estimated 99.8% of all answers being correct. A separate check of the 144 questions that target cause-specific history found 8 direct questions to which the mapper returned unknown although the record settles them (Appendix C), so its rare errors also include withheld history.

In the revision, we made four changes:

  • Conclusion: the limitations now quantify how much derived values contribute to accuracy and to the benefit of video (see our reply to 6aM3-Q1).
  • Conclusion: also, “one LLM derives values, maps questions and grades outcomes”.
  • New Appendix J.4 (“Audit of the History-Question Mapping”) reports the audit of 200 mapped questions and its table.
  • Appendix C: new Appendix C.4 reports accuracy when the chart returns reported entries only, and a new paragraph and table report how the mapper answers the questions that target history specific to the confirmed cause.
AuthorsOpenReview ↗

Response to Reviewer wZXh: Q1

Q1 (summarised). How is the DeepSeek question mapping validated?

Thank you for this question. In the submission we audited the grader and the order matcher (Appendix J) but not the question mapping, so we have now audited it as well.

How the mapping works. Each case has a fixed symptom table of 15–24 history features, each recorded as present or absent. DeepSeek-V4.1-Flash receives the model's questions and this table, and, with the prompt in Appendix H.2, answers yes or no only when the documented features establish that the whole question is true or false, and unknown otherwise.

Audit. To show how reliable the mapping is, we drew a stratified sample of 200 mapped questions from the Table 2 video and blind consultations of all five systems (for each system and condition, 10 questions answered yes and 10 answered unknown), checked each answer against the case's symptom table, and weighted the error rates back to all 8,804 questions. The table below summarises the audit by mapper answer.

Mapper answerShare of all questionsSampledWrongError rate (weighted)
yes8.4%10042.2%
Other answers91.6%10000.0%
All answers100%20040.2%
  • Wrong yes answers: 4 of the 100 sampled yes answers were wrong (2.2% after weighting). In each case the mapper said yes to a feature the record does not contain: a seizure, dizziness, restricted eye movements, and “mainly a stomach problem” for a patient whose main problem was jerks.
  • Wrong unknown answers: none of the 100 sampled unknown answers was wrong, so no sampled answer withheld a feature that the record documents as present.
  • Questions on cause-specific history: a random sample mostly draws questions the record cannot settle, so we also checked the 144 video-condition questions that target history specific to the confirmed cause. For 8 of them the mapper returned unknown to a direct question that the record settles (Appendix C).

We conclude that an estimated 99.8% of the mapper's answers are correct, and that its errors are rare in both directions: it seldom adds a symptom and seldom withholds one.

In the revision, we made three changes:

  • New Appendix J.4 (“Audit of the History-Question Mapping”) describes what the mapper does, how the 200 questions were sampled and checked, the table of error rates above, and the four wrong yes answers.
  • Conclusion of Appendix J.4: “An estimated 99.8% of the mapper's answers are correct. No sampled answer withheld documented history, although a targeted check of the questions on cause-specific history found 8 such cases among 144”.
  • Appendix C: a new paragraph and table report that targeted check: how often the models ask about cause-specific history, and how the mapper answers.
Reviewer_6aM3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces DynamicDx, a benchmark of 71 neurological cases with real patient videos. It finds that videos help models in diagnoses, but the gain primarily comes from helping models order more informative tests and obtain useful evidence, rather than recognizing symptoms from the video.

Critical strengths
  1. Identifying that videos can help diagnoses primarily by motivating further investigation rather than providing visual evidence is a useful insight.
  2. The experiments are thorough, comparing different models on different cases like with video, without video, shuffled frames, and oracle evidence. The replay experiment is very well designed.
Critical weaknesses
  1. The dataset is small and limited. It heavily relies on synthetic information about tests, and most history questions are answered with "unknown." This brings into question the idea of evidence acquisition being the bottleneck.
  2. The authors' claims about the video dynamics not helping at all seem too bold when there are some models that show a statistically significant improvement because of dynamics. Claims like "current models use what a clip shows rather than how it unfolds: shuffling the frames produces no reliable accuracy loss" could be softened, since in some models it does.
  3. Temporal window training does not create a significant improvement in final accuracy. It seems that part of the improvement from literature retrival comes from just expanding the list of possible diagnoses, rather than from retrieving relevant knowledge (since clean and mismatched retrieval perform similarly in Table 29).
Questions for authors
  1. What is the effect of using semi-synthetic data on the model? Is there a subset of cases with enough real tests and known history to compare the performance on real and semi-synthetic data?
9 comments
AuthorsOpenReview ↗

Response to Reviewer 6aM3: W1 (part 1 of 2)

W1 (summarised). The dataset is small and relies heavily on synthetic test results, and most history questions are answered “unknown”. This questions the claim that evidence acquisition is the bottleneck.

Thank you for raising this important point. We would also like to explain why we believe the main conclusion still holds with a relatively small dataset and synthetic test results.

  • Benchmark size limitation due to difficulty of data collection. Each case must combine an authentic patient video, a confirmed diagnosis and a documented work-up, all from the same open-access report. Building one means reading the full paper and summarising its history, examination, investigations and diagnosis by hand. A case report usually describes a single patient, so one paper yields one case, and most case reports do not include a video at all. In total, 66 reports met these requirements and gave 71 cases. Related benchmarks of interactive multimodal diagnosis are of the same order: For video-based diagnosis paper, Multimodal AMIE (Google DeepMind and Google Research; Nature Medicine) uses 105 scenarios (Saab et al., 2026) and the AI co-clinician study (Google DeepMind and Google Research; arXiv preprint) uses 20 (Shah et al., 2026), and both rely on patient actors rather than the videos of real patients that we use (Table 1). This shows our unique contribution to the development of benchmarks for video-based diagnosis.
  • The key evidence is mostly real. Of the 301 decisive entries (the test results that settle a diagnosis), 256 are taken directly from the source reports.
  • The conclusion holds regardless of derived values. To show that the derived values do not change our conclusions, we ran two experiments that remove them while keeping each consultation's questions, answers and orders fixed: (i) replacing the 1,086 derived values that state a specific finding with a neutral one (Appendix C, Table 14), and (ii) withholding every derived entry, so that the chart returns only what the source article reports (Appendix C.4; see our reply to your Q1). We summarise the results for the three conditions in the table below.
ChartBlindVideoOracle evidenceVideo − blind
Original (Table 2)13.830.182.0+16.3
(i) Derived findings neutralised9.319.480.0+10.1
(ii) Reported entries only7.622.377.5+14.6

Accuracy (%), mean of five systems, 71 cases. Row (i) re-runs the diagnosis turn; the same re-run on the original chart gives 14.4, 29.3 and 81.4 (+14.9). Row (ii) applies the change measured in the replay of Q1 to the accuracies of Table 2.

Removing the derived values lowers absolute accuracy in every condition, but the ordering does not change: accuracy with video remains higher than with blind input, and accuracy with oracle evidence remains substantially higher than with video.

AuthorsOpenReview ↗

Response to Reviewer 6aM3: W1 (part 2 of 2)

W1 (summarised). The dataset is small and relies heavily on synthetic test results, and most history questions are answered “unknown”. This questions the claim that evidence acquisition is the bottleneck.

  • Unknown history answers reflect a record limitation, and history has a limited effect. The records settle only 8–33% of the questions models ask (Table 15), so most history questions return unknown. However, among the 144 questions in the video condition that target a history feature specific to the confirmed cause, only 41% return unknown (Appendix C of the revision). Moreover, in clinical practice not every history question receives a definite answer, and a diagnosis must still be reached from partial information. To show that this sparse history does not account for the main result, we ran a replay experiment: for each video consultation, we re-ran the diagnosis turn after replacing either its history or its investigations with those of the blind consultation of the same case (Table 11). The table below summarises the mean accuracy of the four combinations of history and investigations.
History fromInvestigations from: videoInvestigations from: blindVideo − blind
Video29.317.2+12.1 [+6.5, +17.7]
Blind29.016.9+12.1 [+5.9, +18.3]
Video − blind+0.3 [−2.8, +3.1]+0.3 [−1.7, +2.5]

Accuracy (%) of the replayed diagnosis turn, mean of five systems; differences with 95% CIs.

Replacing the history changes accuracy by only 0.3 points, whereas replacing the investigations lowers it by 12.1 points with either history. We conclude that the benefit of video arises from the tests it prompts, rather than from the history it elicits.

  • History answerability does not change the conclusion. To show that the main results do not depend on sparse history, we split the 71 cases at the median answerability (Table 16; Appendix C.5 of the revision). A case's answerability is the share of all questions asked about it, pooled over the five systems and the six conditions of Table 15, that its record settles with a yes or no; it ranges from 5.7% to 28.2%, with a median of 14.7%. We recomputed the two main contrasts in each half, using the runs of Table 2. The table below summarises the results for each half and for all cases.
CasesnVideo − blindOracle − video
Answerability above the median (>14.7%)35+21.7+48.6
Answerability at or below it (≤ 14.7%)36+11.1+55.0
All cases71+16.3+51.8

Difference in accuracy (percentage points), mean of five systems. Oracle: oracle evidence.

The results show that history answerability does not change the conclusion that video improves accuracy over blind input and that oracle evidence improves it far more, although the benefit of video is larger where answerability is higher.

In the revision, we made four changes:

  • Conclusion: the limitations now quantify how much derived values contribute to accuracy and to the benefit of video (see our reply to your Q1).
  • Section 4.2: the text now adds that “withholding all derived entries narrows it from 16.3 to 14.6”, where “it” is the benefit of video over blind input.
  • New Appendix C.4 and C.5: C.4 reports accuracy when the chart returns reported entries only, and C.5 repeats the main comparisons on the cases with mostly real charts and on those with more answerable records.
  • Appendix C: a new paragraph and table report how often the models ask about history specific to the confirmed cause, and how often those questions return unknown.
AuthorsOpenReview ↗

Response to Reviewer 6aM3: W2

W2 (summarised). The claim that temporal dynamics do not help is too strong: some models show a statistically significant ordered-versus-shuffled effect. Statements such as “shuffling the frames produces no reliable accuracy loss” should be softened.

Thank you for this careful reading; we agree that our wording was too strong. Pooled over the five models, ordered frames outperform shuffled frames by 5.4 points (95% CI −0.3 to +11.3), but the effect differs between models (Table 2):

  • MiniMax-M3 and Qwen3.8-flash each lose 9.9 points when the frames are shuffled;
  • Gemma-4-31B loses 5.6 points and GPT-5.6-luna 1.4 points;
  • MiMo-v2.5 is unaffected.

In the revision, every sentence that made the stronger claim has been rewritten (old → new):

  • Abstract: “neither recognition alone nor temporal order explains the gain … shuffling the frames produces no reliable accuracy loss” → “recognition alone does not explain the gain … and frame order matters for some models but not for others”.
  • Figure 1 caption: “not through its dynamics” is removed and “mainly” is added.
  • Introduction: “current models use what a clip shows rather than how it unfolds …” → “the answer depends on the model: shuffling the frames costs two models 9.9 points each and leaves another unaffected”.
  • Second contribution: “rather than through recognition or temporal order” → “rather than through recognition”, and “mainly” is added.
  • Section 4.2: “perform alike, consistent with models that … largely ignore frame order” → “move accuracy less than evidence does, and their effects differ between models”, with the per-model losses added: “by 9.9 points for MiniMax-M3 and Qwen3.8-flash and not at all for MiMo-v2.5”.
  • Conclusion: “shuffling the frames costs little” → “frame order matters for only some models”, in line with the revised abstract.
  • Appendix B: “lack of a reliable ordering effect” → “small pooled ordering effect”.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their careful reading and constructive comments. We reply to each point under the corresponding review, and summarise here what the new analyses show and how the paper has changed (changes are marked in red in the revised PDF).

The main findings hold. To show that the main findings do not depend on the derived chart values or on multiple comparisons, we withheld every derived entry and applied a Holm–Bonferroni correction to all 27 reported contrasts (table below).

FindingEstimate [95% CI]Holm p
Video − blind input+16.3 [+9.3, +23.7]<0.001
Oracle evidence − video+51.8 [+43.4, +60.0]<0.001
Video − blind investigations (replay)+12.1 [+6.5, +17.7]<0.001

Difference in accuracy (percentage points), mean of five systems, 71 cases.

  • Video improves accuracy over blind input, and the gain remains +14.6 points when the chart returns only the entries reported in the source articles (6aM3-Q1).
  • Evidence acquisition is the bottleneck. Oracle evidence raises accuracy far more than video in every system, and by +55.2 points with reported entries only.
  • The gain from video arises from the tests it prompts. In the replay, the video consultation's investigations add 12.1 points and its history 0.3.

New analyses in this rebuttal.

  • Withholding all derived chart entries, and comparing cases whose charts are mostly real or whose records answer more history questions (6aM3-Q1).
  • A Holm–Bonferroni correction over the 27 effects the paper reports, and over 37 once the retrieved-versus-unrelated contrasts are added (Fhy4-Q1).
  • An audit of the history-question mapping (wZXh-Q1), a check of the questions on cause-specific history (6aM3-W1), and a test of whether derived entries can be told apart from reported ones (wZXh-Q2).
  • A release of all 2,838 Table 2 consultations and a clip-cutting script (Fhy4-W3).

Changes to the paper.

  • Abstract and introduction. The benefit of video is given as “16.3 percentage points on average”, frame order is described as mattering “for some models but not for others”, and the describer sentence adds its measured effect, “naming the type of movement in 54% of clips instead of 10%”.
  • Contributions and closing sentence. The closing sentence becomes “what a model sees helps mainly through the tests it prompts”, as the replay and oracle-evidence results in the table above show.
  • Sections 4.2–4.4. The text gives the frame-order effect for each model, reports which effects survive the correction, and states what post-training and retrieval each improve: the description of the sign and the selection of tests.
  • Clinician reference. “Neurologist” is corrected to neurocritical-care physician, Appendix B reports the clinician's background, and Section 4.1 explains why the clinician's and the models' rates are not directly comparable.
  • Limitations. The conclusion now quantifies the contribution of derived values, notes that one LLM performs several steps of the environment, and notes that the clinician reference is a single physician.
  • New appendix analyses. Appendix C.4 (charts with reported entries only), C.5 (cases with mostly real charts or more answerable records), C.6 (distinguishing derived from reported entries), a new paragraph of Appendix C on cause-specific history, J.4 (audit of the history-question mapping) and Appendix K (Holm-adjusted p-values).
AuthorsOpenReview ↗

Request for assistance with reviewer follow-up

Dear Area Chair,

Thank you for coordinating the discussion. We have responded to the reviewers’ comments and posted follow-ups. If appropriate, could you kindly remind the reviewers who have not yet responded to our rebuttal to share any remaining concerns before the discussion closes? We would be happy to provide further clarification.

Thank you for your time and help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.