- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CariesBench, a benchmark for evaluating dental caries assessment from standardized multi-view renderings of 3D intraoral scans. It contains 5,932 scans from 2,966 patients, 85,455 rendered images, and 133,994 VQA instances across 19 question types covering arch-level clinical screening, tooth-level diagnostic reasoning, and structured clinical decisions. The construction pipeline separates structured clinical annotations from LLM-based language realization and includes automated auditing and expert validation. Evaluation uses patient-disjoint training, validation, and test partitions. The authors benchmark 24 general-purpose and medical/dental MLLMs and report substantially lower performance on tooth- and surface-level tasks than on arch-level screening. They also fine-tune Qwen3-VL-8B-Instruct with LoRA to obtain IntraoralGPT, reporting an increase in the 19-task macro-average accuracy from 52.7% to 69.5%. Fine-grained surface assessment remains challenging, with the resulting TDR Overall score reaching 44.7%.
Critical strengths
S1: The benchmark connects arch-, tooth-, and surface-level assessment through views rendered from the same underlying anatomy. Its combination of surface-specific clinical states, complementary observations, and multiple question formats provides a useful resource for studying dental MLLMs beyond image-level recognition.
S2: The patient split is applied before downstream sampling and training-set rebalancing, and the construction explicitly separates clinical labels from LLM-generated prose. The reported 20% expert validation, double assessment of 800 instances, and evaluation of 24 baseline models strengthen the empirical foundation. The same-backbone fine-tuning comparison also provides evidence that the benchmark supervision is useful within the reported setting.
Critical weaknesses
W1: The definition and distribution of Sound Tooth Verification (SV) are not sufficiently consistent to establish the validity of this subtask. Appendix F.2 (p. 18) states that non-sound teeth are used for SA/FS, whereas sound teeth are used for SV. Taken literally for the test set, this makes the SV target constant: always predicting “sound” would achieve 100% under the binary metric defined in Appendix K.1. However, Fig. 2(e) distributes SV instances across L0–L7, with only 24.0% assigned to L0. It is unclear whether the construction description is incomplete or the figure uses a different label definition. This matters because SV contributes one-third of TDR Overall and supports the comparison with SA/FS in Finding 1. The actual test-target counts, their relation to the plotted labels, and a constant/majority-answer baseline are needed to resolve this ambiguity.
W2: The experiments do not sufficiently distinguish the observed task difficulty gap from the specific explanation in terms of anatomical localization and cross-view integration. Finding 2 reports a 29.1-point difference between mean ACS and TDR scores. As the authors acknowledge, these groups differ in task format as well as anatomical granularity; Table 8 and Appendix K also show different prediction targets and scoring units. This supports a descriptive performance gap on the constructed tasks, but does not isolate its source. The paper does not report a comparison that holds the clinical target and answer format fixed while varying the available views, or an input ablation establishing dependence on cross-view evidence. The five-step response protocol and final-label accuracy do not independently establish such dependence. A limited paired analysis on existing test cases, or a more restricted interpretation of the findings, would materially strengthen the benchmark's central scientific claim.
Questions for authors
Q1: Does the SV test set contain both sound and non-sound teeth? Please reconcile Appendix F.2 with Fig. 2(e), specifying what the L0–L7 labels in that plot represent and reporting the actual binary target counts for SV in each split. What does an always-sound or majority-answer predictor achieve, and would correcting the description or construction change the reported TDR and aggregate results?
Q2: What evidence separates a limitation in cross-view integration from differences in task definition and answer difficulty? On a limited subset of existing held-out targets, can the authors compare a relevant single view with all views while holding the clinical target, label space, and answer format fixed? Alternatively, please clarify which conclusions are intended only as descriptive comparisons across the current tasks. Such evidence or clarification would affect my assessment of the claimed reasoning insight.
7 comments
Response to Reviewer bTTF
Table R1. L0--L7 Clinical-status Distribution in the Full TDR-SV Benchmark Set
| Clinical status | Number of instances | Percentage (%) |
|---|---|---|
| L0 | 2,425 | 23.97 |
| L1 | 1,444 | 14.27 |
| L2 | 677 | 6.69 |
| L3 | 1,857 | 18.36 |
| L4 | 1,111 | 10.98 |
| L5 | 777 | 7.68 |
| L6 | 1,090 | 10.78 |
| L7 | 735 | 7.27 |
| Total | 10,116 | 100.00 |
For the task-specific SV splits, the actual binary target counts are reported in Table R2.
Table R2. SV Binary Class Distribution Before and After Training-set Rebalancing
| Data partition / stage | Sound (L0) | Non-sound (L1--L7) | Total |
|---|---|---|---|
| Pre-rebalancing training pool | 953 (11.96%) | 7,016 (88.04%) | 7,969 |
| Training (after rebalancing) | 1,550 (25.66%) | 4,491 (74.34%) | 6,041 |
| Validation | 735 (68.56%) | 337 (31.44%) | 1,072 |
| Test | 737 (68.56%) | 338 (31.44%) | 1,075 |
| Post-rebalancing task-specific splits | 3,022 (36.91%) | 5,166 (63.09%) | 8,188 |
We agree that overall accuracy alone is insufficient for evaluating SV under the observed test-set class distribution. We therefore report an always-sound baseline using Accuracy, Macro-F1, Balanced Accuracy, and non-sound recall, as shown in Table R3. The test split contains 737 Sound and 338 Non-sound instances among 1,075 samples. Since sound teeth form the majority class, the always-sound predictor corresponds to the majority-class baseline. It achieves 68.56% Accuracy, 40.67% Macro-F1, 50.00% Balanced Accuracy, and 0.00% non-sound recall. This baseline highlights why accuracy should be interpreted alongside class-sensitive metrics.
Table R3. Always-Sound Baseline on the SV Test Set
| Metric | Always-sound baseline |
|---|---|
| Sound instances | 737 |
| Non-sound instances | 338 |
| Total test instances | 1,075 |
| Accuracy (%) | 68.56 |
| Macro-F1 (%) | 40.67 |
| Balanced Accuracy (%) | 50.00 |
| Non-sound Recall (%) | 0.00 |
The clarification documents the finalized SV construction used in all reported experiments and does not introduce any post-submission changes to the benchmark or evaluation protocol. The evaluation targets and prediction records remain unchanged. Consequently, the reported TDR Overall and 19-task macro-average results are unaffected. The additional class-distribution statistics and always-sound baseline provide further context for interpreting the existing evaluation results.
Response to Reviewer bTTF
W2: Insufficient Evidence Separating Task Difficulty from Anatomical Localization and Cross-view Integration. The experiments do not sufficiently distinguish the observed task difficulty gap from the specific explanation in terms of anatomical localization and cross-view integration. Finding 2 reports a 29.1-point difference between mean ACS and TDR scores. As the authors acknowledge, these groups differ in task format as well as anatomical granularity; Table 8 and Appendix K also show different prediction targets and scoring units. This supports a descriptive performance gap on the constructed tasks, but does not isolate its source. The paper does not report a comparison that holds the clinical target and answer format fixed while varying the available views, or an input ablation establishing dependence on cross-view evidence. The five-step response protocol and final-label accuracy do not independently establish such dependence. A limited paired analysis on existing test cases, or a more restricted interpretation of the findings, would materially strengthen the benchmark's central scientific claim.
Q2: Controlled Single-view versus Multi-view Comparison. What evidence separates a limitation in cross-view integration from differences in task definition and answer difficulty? On a limited subset of existing held-out targets, can the authors compare a relevant single-view with all views while holding the clinical target, label space, and answer format fixed? Alternatively, please clarify which conclusions are intended only as descriptive comparisons across the current tasks. Such evidence or clarification would affect my assessment of the claimed reasoning insight.
A2: We thank you for highlighting this important issue. We agree that the ACS--TDR performance gap is a descriptive benchmark-level disparity and cannot be attributed solely to anatomical granularity, cross-view integration, or any other individual factor. Across the 24 baseline MLLMs, mean ACS Overall accuracy was 67.3%, compared with 38.2% for TDR Overall, corresponding to a descriptive benchmark-level performance difference of 29.1 percentage points. Because ACS and TDR differ in anatomical scope, prediction targets, diagnostic complexity, answer formats, and scoring units, this difference cannot identify the specific source of the performance gap. We have therefore revised Finding 2 to describe this gap as a benchmark-level performance disparity, without attributing it to any single underlying factor or presenting it as a causal effect of anatomical granularity.
To address your request for a controlled single-view versus multi-view comparison, we conducted additional within-task ablations on TDR-FS and ACS-SC using their respective existing patient-disjoint held-out test sets. Within each task, all input conditions were evaluated using the same clinical targets, label spaces, answer formats, prompting protocols, image-resolution settings, model configurations, and deterministic decoding settings. Only the visual input condition was varied. For the repeated-view control, Panel 1 was repeated for TDR-FS and View 5 for ACS-SC, with the number of visual input slots matched to that of the full multi-view condition. These comparisons allow us to examine whether the observed performance differences are associated with access to distinct visual observations under a matched input budget, rather than simply an increased number of visual input slots.
The controlled comparisons are summarized in Table R4 and R5.
Table R4. Comprehensive multi-view and view-wise ablation on TDR-FS
| Input condition | Accuracy (%) | Macro-F1 (%) | Balanced Acc. (%) | (pp) | 95% CI (pp) |
|---|---|---|---|---|---|
| Full multi-view | 29.1 | 16.7 | 28.1 | Ref. | -- |
| Panel 1 | 23.5 | 11.6 | 20.8 | 5.6 | [3.9, 7.7] |
| Panel 2 | 23.0 | 10.0 | 18.6 | 6.1 | [4.4, 8.2] |
| Panel 3 | 23.2 | 9.9 | 18.7 | 5.9 | [4.2, 7.8] |
| Panel 4 | 22.3 | 8.2 | 17.3 | 6.8 | [5.0, 8.8] |
| Panel 5 | 22.1 | 8.4 | 17.4 | 7.0 | [5.3, 9.0] |
| Repeated Panel 1 (equal budget) | 23.2 | 10.6 | 19.4 | 5.9 | [4.1, 7.8] |
| Mean single-view | 22.8 | 9.6 | 18.6 | 6.3 | [4.7, 8.0] |
On TDR-FS, full multi-view input achieved 29.1% Accuracy, 16.7% Macro-F1, and 28.1% Balanced Accuracy. Its Accuracy exceeded the highest-accuracy fixed single-view condition (Panel 1, 23.5%) by 5.6 percentage points, the mean single-view baseline (22.8%) by 6.3 percentage points, and the equal-budget repeated-Panel-1 control (23.2%) by 5.9 percentage points. The paired 95% CIs excluded zero for all five individual panels, the repeated-view control, and the mean single-view baseline.
Response to Reviewer bTTF
We thank you for the careful assessment of CariesBench. We particularly appreciate your recognition of the benchmark’s hierarchical design, patient-level partitioning, clinical annotations, and expert quality control. We address two concerns regarding the finalized SV task construction and the interpretation of the cross-task performance disparity. We clarify the finalized SV construction and label mapping, report the full-benchmark L0--L7 distribution and the binary class distributions before and after training-set rebalancing, provide an always-sound baseline, and present controlled within-task multi-view ablations. We also distinguish the descriptive performance gap across task families from the evidence provided by controlled multi-view comparisons.
A1: We thank you for identifying this inconsistency. An intermediate version of our SV construction script contained a sound-only eligibility condition. This condition was removed before submission, and the finalized SV construction allows both sound and non-sound teeth to be included. However, Appendix F.2 inadvertently retained an outdated description from the intermediate construction protocol, stating that non-sound teeth were used for SA/FS whereas only sound teeth were used for SV. This statement did not describe the finalized benchmark. All reported experiments were conducted using the finalized benchmark containing both sound and non-sound SV instances. Thus, the inconsistency was confined to the outdated appendix description and did not affect the evaluation data or reported results. We apologize for this oversight and have removed the superseded SV-specific eligibility statement from Appendix F.2, while retaining the shared clinical annotation framework across SA, FS, and SV.
In the finalized benchmark, SA, FS, and SV are based on the same L0--L7 clinical-status taxonomy, derived from structured surface-level annotations. Their differences lie in anatomical scope and prediction format, rather than in the underlying clinical annotation system. For SV, the tooth-level binary target is derived from the relevant surface annotations: a tooth is classified as Sound (L0) if all its relevant annotated surfaces are L0, and as Non-sound (L1--L7) if at least one relevant annotated surface has a clinical status from L1--L7. Thus, L0--L7 represent the underlying clinical-status categories, whereas the final SV prediction space is binary (Sound/Non-sound). The L0--L7 distribution reported in Figure 2(e) therefore describes the underlying clinical-status composition of the full TDR-SV benchmark set.
The full TDR-SV Benchmark set contains 10,116 instances, including 2,425 L0 and 7,691 L1--L7 instances. Figure 2(e) shows the L0--L7 clinical-status distribution of the full TDR-SV Benchmark set. Before training-set rebalancing, the full benchmark is partitioned into a training pool of 7,969 instances (953 Sound and 7,016 Non-sound), together with 1,072 validation instances (735 Sound and 337 Non-sound) and 1,075 test instances (737 Sound and 338 Non-sound). After task-specific training-set rebalancing, the training split contains 6,041 instances (1,550 Sound and 4,491 Non-sound), while validation and test remain unchanged. The post-rebalancing task-specific splits therefore contain 8,188 instances, comprising 3,022 Sound and 5,166 Non-sound instances.