acceptodds
Withdrawn from ICLR 2027

OralFlow: A Longitudinal Multimodal Benchmark across Clinical Workflows in Dentistry

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have expanded dental AI from isolated image interpretation to open-ended clinical interaction. However, existing dental models often frame diagnosis as a static, modality-specific problem. Complete dental workflows require integrating complementary multimodal evidence and continuously retaining, retrieving, and updating evidence across the typical stages of dental care, including diagnosis, treatment planning, and follow-up. To bridge this gap, we introduce OralFlow-Bench, the first benchmark for evaluating longitudinal multimodal dental clinical workflows. It contains 668 cases from two complementary sources: OralFlow-Literature comprises 281 published long-horizon case reports spanning initial presentation, treatment, and follow-up, while OralFlow-Cohort comprises 387 complex, multi-visit hospital cases with 10,326 images across six modalities. We evaluate six MLLMs with six memory methods across five task categories to handle long-context clinical evidence in longitudinal clinical workflows. On end-to-end evaluation across the clinical workflow, models achieve 37.56% diagnosis accuracy and 29.57% treatment and 40.64% follow-up scores, revealing the challenges in handling complete longitudinal dental workflows. Even given ground-truth observations, memory methods yield variation in diagnostic accuracy across MLLMs, ranging from 46.94% to 93.88%, and these differences propagate to subsequent stages, further degrading treatment and follow-up performance. Missing-modality ablations reveal the complementary roles of multiview skeletal model and X-Ray, while noise ablations show that robust performance depends on selective memory updating and retrieval rather than context size alone. Code and benchmark resources are available at https://anonymous.4open.science/r/OralFlow/.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Longitudinal Clinical Workflows, Dental AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CabhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OralFlow Bench, a benchmark for longitudinal multimodal dental clinical workflows. It combines 281 published long-horizon case reports spanning initial presentation, treatment, and follow up (OralFlow Literature) with 387 complex multi-visit hospital cases covering 10,326 images across six modalities (OralFlow Cohort). Six MLLMs are evaluated with six memory methods across five task categories under staged evidence release that mirrors real clinical decision making, with GPT-5-mini as the scoring judge validated against dentist judgments. End to end, models reach 37.56% diagnosis accuracy, with the gap widening at treatment and follow up stages.

Critical strengths

The workflow framing is the paper's real contribution and it is a good one. Instead of the standard static context benchmark, OralFlow stages evidence release across diagnosis, treatment planning, and follow up, so models must continuously retain, retrieve, and update clinical evidence the way a care team actually does. This is, to my knowledge, the first benchmark to evaluate multimodal clinical memory this way, and the design choices are careful: a task never observes its own reference answer, and answers are generated only from information available at the scheduled stage, which cleanly separates memory failures from perception failures. The controlled finding that memory methods vary from 46.94% to 93.88% diagnostic accuracy even given ground truth observations, and that these differences propagate to downstream stages, is genuinely informative and not something any existing benchmark could have surfaced.

The empirical scope is unusually thorough for a first benchmark paper. Two complementary cohorts (literature derived and hospital derived), six modalities, follow up windows from 0.5 to 43 years, six frontier and medical-specialized MLLMs, and six representative memory methods give the study real coverage, and the missing modality and noise ablations convert raw failure numbers into practical guidance, most notably that robustness depends on selective memory updating and retrieval rather than context size alone. The evaluator methodology is also more careful than the norm: the GPT-5-mini judge is validated against dentist judgments with confusion matrices, and a five run stability analysis reports 90% to 98.6% exact agreement depending on the model being scored. Most benchmark papers ship an unvalidated LLM judge; this one at least measures its evaluator. The headline negative result, that verified observations support diagnosis far better than treatment and follow up planning, is exactly the kind of clinically meaningful gap a benchmark should expose.

Critical weaknesses

The main issues are about verifiability and must be resolved at camera ready, but none of them undermines the scientific content. First, the ethics statement names the institution and IRB number only as placeholders (XXX hospital, IRB No. XX-XX-XX-X), and the hospital cohort is not releasable. I understand the double blind constraint and patient privacy obligations, but a benchmark whose differentiating cohort cannot be independently checked lives or dies on trust. The authors should provide the de anonymized approval record to the program chairs and specify a controlled access mechanism for OralFlow Cohort, since a longitudinal dental workflow benchmark that the community can never touch has limited lasting value. Second, the human validation of the GPT-5-mini judge covers 40 cases, which is small relative to the benchmark, and the appendix confusion matrices show non-trivial judge dentist disagreement in at least one task family; the stability analysis mitigates randomness but not systematic judge bias, so scaling the human validation should be a condition of the camera ready. Third, the cohort is single center, which limits demographic and practice-style generalization, and the licensing status of republishing images and text from published case reports in the Literature cohort is never discussed. These are fixable. The staged workflow design, the decoupling of perception from memory, and the ablation methodology are the contributions, and they stand.

Questions for authors
  1. What controlled access or evaluation-as-a-service mechanism do you plan for OralFlow Cohort, and can the de anonymized IRB approval be shared with the program chairs during review?
  2. Can the dentist-judge validation be scaled beyond 40 cases, ideally per task family, so that treatment and follow up rubric scores carry quantified judge reliability?
  3. For the Literature cohort, what is the copyright and consent status of republishing case report images and text, and will derived data be released with documentation?
Reviewer_g9KaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OralFlow, a longitudinal multimodal benchmark for dental clinical workflows. It contains 387 single-center hospital cases (complex dentofacial deformities) and 281 public literature cases, for 668 cases and 12,549 evaluation tasks in total. The construction pipeline is: case timeline → atomic evidence and evidence graph → five task categories (diagnosis, treatment, follow-up, etc.) → answering under different memory methods → LLM-judge scoring. The authors evaluate 6 models and 6 memory methods, and run modality-missing and noise ablations. They conclude that current models remain substantially limited in end-to-end clinical workflows.

Critical strengths
  1. The topic is valuable. The paper moves dental multimodal evaluation from single-shot image QA to staged workflows, and it explicitly specifies when evidence becomes visible, when questions are asked, and when gold answers are released. This is more rigorous than generating QA directly from whole case reports.

  2. Using both expert-standard observations and model-generated observations makes it possible to isolate the effect of perception errors on downstream tasks.

  3. The appendix provides case examples, scoring prompts, and a memory cost analysis, which gives good transparency.

Critical weaknesses
  1. Ablation baselines are inconsistent with the main table. Figure 4 uses OralGPT-Omni-7B with Vector/Summary/Full Context memory and shows a standard baseline of roughly 46% Diagnosis Accuracy and 64% ERS. The mean of the same three configurations in Table 1 is about 77.75% and 87.43%. The zero-noise condition in Figure 5 likewise does not match. The paper does not state whether the ablations use a different case subset, task set, or configuration. Until this is resolved, the size of the drops caused by missing modalities and noise cannot be verified, and the ablation conclusions cannot be reproduced.

  2. The clinical validity of the scoring rules is questionable. The scoring prompts (Fig. 28/29) give “skeletal Class III” and “bilateral Class III molar relationship” as an equivalence example. The former describes the jaw (skeletal) relationship and the latter the dental (occlusal) relationship; they are related but not equivalent. Angular measurements uniformly allow a ±10° tolerance with no per-metric justification (e.g., for a case with ANB ≈ −7°, −1° would be judged correct). This may inflate Accuracy/ERS and change model rankings. Moreover, judge validation covers only one model combination (OralGPT + Vector) on 40 cases, and the treatment Rubric comparison is made only on means (physician 40.99 vs. GPT-5-mini 36.00), which does not demonstrate per-item agreement.

  3. Visible observations are not cleanly isolated from hidden decisions, and the reference answers have quality problems. In Case 1, the S4 CT observation answer (A8) already contains treatment directions such as mandibular setback, maxillary advancement, and derotation, which contradicts the stated design that treatment decisions are kept in hidden records. The same case also has an internal inconsistency: page 28 records a 1 mm anterior open bite, while the comprehensive diagnosis on page 31 states “Overbite is not assessable.” Treatment reference answers are based on the “treatment actually adopted in the source case,” which introduces circular reasoning (e.g., justifying “active treatment is needed” by the fact that a complete orthognathic pathway was planned), and reasonable but different plans may be penalized. High scores on the standard trajectory therefore do not necessarily reflect independent clinical integration by the model.

  4. The protocol measures “streaming QA with gold-answer feedback,” and the conclusions are overstated. The memory update in Eq. 4 writes the reference answers of earlier tasks into memory, so a model that answered incorrectly early on receives the gold answer in later stages. The claims of “end-to-end clinical workflow” and “error cascade propagation” are therefore insufficiently supported. In addition, some tasks do not match their category names (e.g., “follow-up planning” actually asks about previously performed procedural details, and “memory update” only asks about a later biopsy result), and question-only / no-memory baselines are missing.

Questions for authors
  1. Please explain the discrepancy between Figures 4/5 and Table 1, and provide the case IDs, task denominators, and raw scores used in the ablations.
  2. Could orthodontic experts redefine the concept-equivalence and per-metric tolerance rules, re-score the affected results, and report per-item MAE/ICC for the judge together with its effect on model rankings?
  3. Could the authors audit the visible VQA for diagnostic inferences and treatment suggestions, add an “objective observations only” version, and have independent experts who are blind to the source case’s actual treatment provide a set of acceptable treatment plans?
  4. Could the authors add three memory-write controls (new observations only / the model’s own prior answers / gold answers), as well as question-only and no-memory baselines?
Reviewer_FPzHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

OralFlow-Bench evaluates how multimodal models handle patient information over time in dental care. It includes 668 cases and 12,549 tasks covering diagnosis, treatment planning, and follow-up. The authors build clinical timelines and evidence graphs, with automated checks and dentist review to improve data quality. Experiments compare six models and six memory methods. Results show that image understanding errors can seriously affect later clinical decisions. However, it remains unclear whether the benchmark mainly tests long-term memory or general clinical reasoning.

Critical strengths
  1. The paper addresses an important problem in clinical AI by evaluating how multimodal models handle patient information across different stages of care. This goes beyond isolated image understanding or single-turn clinical questions.

  2. The benchmark construction is carefully designed. It combines clinical records and published case reports to improve question quality and reduce information leakage.

  3. The experiments cover several multimodal models and memory methods. The analysis of perception errors is particularly useful, showing how mistakes in understanding clinical images can affect later reasoning and decisions.

Critical weaknesses
  1. The benchmark addresses an important clinical application, but its broader value to the ML community is not fully clear. Long-term memory, multimodal reasoning, and error propagation have already been studied in other settings. The paper should better explain what new ML challenges OralFlow reveals and why existing benchmarks cannot capture them. The current findings are useful, but their implications for developing better general-purpose models remain limited.

  2. The experiments test several models and memory methods, but it is still unclear whether the benchmark really measures long-term memory or just general clinical reasoning. Most results also use verified observations, while the end-to-end tests cover fewer settings. Some simple comparisons are missing. For example, the authors could test full history against only recent observations or shuffled histories. It would also help to see how performance changes as the time gap or number of stages increases. Without these tests, poor results may simply come from difficult clinical questions rather than weak long-term memory.

  3. OralFlow-Bench focuses on selected complex dental cases, but it is unclear how well this population represents broader clinical practice. The paper provides limited evidence across patient demographics, disease types, severity levels, and healthcare settings. This raises concerns about how broadly the results can be applied. More importantly, the paper does not clearly show what new ML challenges this benchmark reveals beyond existing multimodal and long-term memory benchmarks. Without stronger evidence of generalizable insights, the contribution seems more relevant to a specialized clinical AI venue than to a general ML conference like ICLR.

Questions for authors
  1. The benchmark includes 668 cases from specialized clinical sources. What makes this dataset particularly useful for the broader ML community, like ICLR? Can the authors show that it reveals new failures in multimodal reasoning or long-term memory that existing benchmarks cannot capture? Without such evidence, it is difficult to see the contribution beyond a domain-specific clinical evaluation.

  2. How representative are the selected cases of broader dental practice? Can the authors provide patient demographics, disease distributions, and severity levels, and clarify which clinical populations the benchmark is intended to represent?

  3. What new challenges does OralFlow-Bench reveal for the broader ML community that existing multimodal reasoning and long-term memory benchmarks do not capture? Can the authors support this with specific experimental findings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.