- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DynamicDx is a process-level benchmark for video-based diagnosis: 71 real patient videos across 11 neurological sign categories, each paired with the confirmed diagnosis and a fixed chart built from the source case report. Stage 1 tests reactive perception (describe the sign, propose a differential); Stage 2 tests proactive evidence acquisition (history questions, investigation orders, final diagnosis) against the fixed chart. Five VLMs are compared under controlled conditions (blind, single frame, shuffled, own words, clinician reference, oracle evidence), plus two interventions: a post-trained temporal-window describer and source-clean literature retrieval. The headline finding is that video helps mainly through the investigations it prompts, not through recognition or temporal order.
Critical strengths
DynamicDx links real patient videos to confirmed diagnoses and a fixed chart from the same case report, and it scores recognition, history taking, test ordering, and final diagnosis separately. That makes it possible to see where a model fails, not just whether the final answer is right.
The second strength is that the central claim rests on more than a single accuracy number. The paper shows that video mainly helps through the tests it prompts, and the trajectory replay and oracle evidence results point the same way.
Critical weaknesses
The evidence base is small and partly synthetic. The benchmark has only 71 cases, with one category containing a single clip and several categories below 10 cases. More importantly, 5,972 of 6,398 chart entries are derived rather than reported, and the paper itself shows that removing derived values lowers accuracy by about ten points. This makes absolute accuracy and some contrasts difficult to interpret as measures of real diagnostic behavior.
The main empirical pattern is mixed, but the narrative is stronger than the evidence. Multi-turn consultation improves three models and degrades two; ordered video exceeds shuffled frames by only 5.4 points with an interval including zero (-0.3, +11.3); own words change accuracy by -2.8 to +1.4 points; and retrieval gains remain only for MiMo-v2.5 and MiniMax-M3. The conclusion that “seeing better helps when it leads to asking better” is plausible but not established at the claimed strength.
Reproducibility is limited. The pipeline depends on several proprietary or rapidly changing models, including GPT-5.6-luna, DeepSeek-V4.1-Flash, and Qwen3.8-flash, and no code or model release is described. The source videos are not redistributed for licensing reasons, which further limits independent reproduction.
Questions for authors
Which conclusions survive a correction for multiple comparisons, given that several key intervals include zero?
7 comments
Response to Reviewer Fhy4: W2
W2 (summarised). The narrative is stronger than the evidence: multi-turn helps three models and hurts two; ordered minus shuffled is 5.4 points with an interval including zero ([−0.3, +11.3]); own words change accuracy by −2.8 to +1.4; retrieval gains hold only for MiMo-v2.5 and MiniMax-M3. “Seeing better helps when it leads to asking better” is plausible but not established at the claimed strength.
Thank you; we agree that parts of the narrative were stated more strongly than the evidence supports. We have rewritten each of the four points you list, and the closing sentence. We also explain which results the central claim rests on.
What we changed (old → new).
- Frame order: “shuffling the frames produces no reliable accuracy loss” → “frame order matters for some models but not for others”, with the loss of each model given in Section 4.2 (see our reply to 6aM3-W2).
- Multi-turn consultation: Section 4.2 no longer says that the conditions “perform alike”; it says that their “effects differ between models”. Multi-turn consultation helps GPT-5.6-luna (+5.6 points), MiniMax-M3 (+8.4) and Qwen3.8-flash (+1.4), and hurts Gemma-4-31B (−7.0) and MiMo-v2.5 (−2.8).
- Own words: the −2.8 to +1.4 point range is now called “no detectable difference” rather than equivalence between the two inputs.
- Retrieval: Section 4.4 now states that “the MiMo-v2.5 gain survives Holm correction” (Holm-adjusted p = 0.016; for MiniMax-M3, p = 0.16), and that the retrieved content itself improves accuracy only for MiMo-v2.5 and test selection for MiMo-v2.5 and Qwen3.8-flash (see our reply to 6aM3-W4).
- Closing sentence: “seeing better helps when it leads to asking better” → “what a model sees helps mainly through the tests it prompts”.
What the central claim rests on. The new closing sentence follows from two results, not from the overall pattern. Both have confidence intervals that exclude zero, and both survive a Holm–Bonferroni correction over all 27 effects the paper reports, listed in our reply to Fhy4-Q1 (Holm p ≤ 0.003).
- Replay: the investigations from the video consultation add 12.1 points (95% CI 6.5 to 17.7), while its history adds only 0.3 (Table 11).
- Oracle evidence: supplying the decisive investigations raises accuracy by 39.4–59.2 points over video, in each of the five systems.
Response to Reviewer Fhy4: W3
W3 (summarised). Reproducibility is limited: the pipeline depends on proprietary or fast-changing models (GPT-5.6-luna, DeepSeek-V4.1-Flash, Qwen3.8-flash), no code or model release is described, and the source videos are not redistributed.
Thank you for raising reproducibility; we may not have made the release visible enough. The data, prompts and evaluation code are available at https://anonymous.4open.science/r/dynamicDx-53C8 (Reproducibility statement). The README gives the exact command for every condition, and every prompt is reproduced in full in Appendix H. In the revision, the conclusion also points to this release.
- Videos. 18 of the 66 source articles use the CC BY-NC-ND licence, which does not allow edited copies to be shared, so we cannot distribute the trimmed clips. Instead, the repository includes a script that downloads each article's original videos from Europe PMC and cuts every clip from them.
data/clips.jsonrecords each clip's source file and start and end time, and, for the eight clips taken from split-screen or two-view recordings, the crop box; re-cutting all 71 clips this way reproduces the benchmark clips up to re-encoding differences. - Open-weight models. Of the seven models in the pipeline, six have openly released weights. Among the five evaluated systems these are Gemma-4-31B, MiniMax-M3 and MiMo-v2.5, and Qwen3.8-flash, whose weights are released as Qwen3.8-Flash-Next. The other two are DeepSeek-V4.1-Flash, which grades outcomes and maps questions and orders, and Qwen3.5-4B, the describer we post-train. We ran the evaluated systems through hosted endpoints, and the evaluation code accepts any model served through the same API, so the benchmark can be re-run on locally served weights.
- GPT-5.6-luna is the only model without open weights. Evaluating a hosted frontier model is common in this area: CRAFT-MD (Harvard and Stanford; Nature Medicine; Johri et al., 2025) and AgentClinic (Johns Hopkins and Stanford; npj Digital Medicine; Schmidgall et al., 2026) evaluate hosted models such as GPT-4, and AMIE (Google; Nature; Tu et al., 2025; McDuff et al., 2025) and Multimodal AMIE (Google; Nature Medicine; Saab et al., 2026) are themselves systems whose weights are not released.
- Saved outputs. Every call uses a fixed provider endpoint and temperature 0. Because hosted models change over time, the repository also includes the model outputs behind Table 2 (
runs/): all 2,838 consultations of the eight conditions for the five systems, each with its questions, answers, orders, released chart entries and final diagnosis. The released grader can re-grade them without calling any model.
Response to Reviewer Fhy4: W1
Thank you for raising this important point. We agree that 71 cases is a small sample and that most chart entries are derived rather than reported, and the revision states both limits more clearly. We would also like to explain why we believe the main conclusion still holds.
Accuracy (%), mean of five systems, 71 cases. Row (i) re-runs the diagnosis turn; the same re-run on the original chart gives 14.4, 29.3 and 81.4 (+14.9). Row (ii) applies the change measured in the replay of 6aM3-Q1 to the accuracies of Table 2.
Removing the derived values lowers absolute accuracy in every condition, but the ordering does not change: accuracy with video remains higher than with blind input, and accuracy with oracle evidence remains substantially higher than with video.
In the revision, we made three changes: