- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces OralFlow Bench, a benchmark for longitudinal multimodal dental clinical workflows. It combines 281 published long-horizon case reports spanning initial presentation, treatment, and follow up (OralFlow Literature) with 387 complex multi-visit hospital cases covering 10,326 images across six modalities (OralFlow Cohort). Six MLLMs are evaluated with six memory methods across five task categories under staged evidence release that mirrors real clinical decision making, with GPT-5-mini as the scoring judge validated against dentist judgments. End to end, models reach 37.56% diagnosis accuracy, with the gap widening at treatment and follow up stages.
Critical strengths
The workflow framing is the paper's real contribution and it is a good one. Instead of the standard static context benchmark, OralFlow stages evidence release across diagnosis, treatment planning, and follow up, so models must continuously retain, retrieve, and update clinical evidence the way a care team actually does. This is, to my knowledge, the first benchmark to evaluate multimodal clinical memory this way, and the design choices are careful: a task never observes its own reference answer, and answers are generated only from information available at the scheduled stage, which cleanly separates memory failures from perception failures. The controlled finding that memory methods vary from 46.94% to 93.88% diagnostic accuracy even given ground truth observations, and that these differences propagate to downstream stages, is genuinely informative and not something any existing benchmark could have surfaced.
The empirical scope is unusually thorough for a first benchmark paper. Two complementary cohorts (literature derived and hospital derived), six modalities, follow up windows from 0.5 to 43 years, six frontier and medical-specialized MLLMs, and six representative memory methods give the study real coverage, and the missing modality and noise ablations convert raw failure numbers into practical guidance, most notably that robustness depends on selective memory updating and retrieval rather than context size alone. The evaluator methodology is also more careful than the norm: the GPT-5-mini judge is validated against dentist judgments with confusion matrices, and a five run stability analysis reports 90% to 98.6% exact agreement depending on the model being scored. Most benchmark papers ship an unvalidated LLM judge; this one at least measures its evaluator. The headline negative result, that verified observations support diagnosis far better than treatment and follow up planning, is exactly the kind of clinically meaningful gap a benchmark should expose.
Critical weaknesses
The main issues are about verifiability and must be resolved at camera ready, but none of them undermines the scientific content. First, the ethics statement names the institution and IRB number only as placeholders (XXX hospital, IRB No. XX-XX-XX-X), and the hospital cohort is not releasable. I understand the double blind constraint and patient privacy obligations, but a benchmark whose differentiating cohort cannot be independently checked lives or dies on trust. The authors should provide the de anonymized approval record to the program chairs and specify a controlled access mechanism for OralFlow Cohort, since a longitudinal dental workflow benchmark that the community can never touch has limited lasting value. Second, the human validation of the GPT-5-mini judge covers 40 cases, which is small relative to the benchmark, and the appendix confusion matrices show non-trivial judge dentist disagreement in at least one task family; the stability analysis mitigates randomness but not systematic judge bias, so scaling the human validation should be a condition of the camera ready. Third, the cohort is single center, which limits demographic and practice-style generalization, and the licensing status of republishing images and text from published case reports in the Literature cohort is never discussed. These are fixable. The staged workflow design, the decoupling of perception from memory, and the ablation methodology are the contributions, and they stand.
Questions for authors
- What controlled access or evaluation-as-a-service mechanism do you plan for OralFlow Cohort, and can the de anonymized IRB approval be shared with the program chairs during review?
- Can the dentist-judge validation be scaled beyond 40 cases, ideally per task family, so that treatment and follow up rubric scores carry quantified judge reliability?
- For the Literature cohort, what is the copyright and consent status of republishing case report images and text, and will derived data be released with documentation?