acceptodds
Withdrawn from ICLR 2027

Pocket-Dentist: Benchmarking Compact Vision-Language Models for Dental Image Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Oral diseases affect approximately 3.7 billion people, yet access to specialist screening remains limited for many people. As smartphones become more widely available, compact vision-language models (VLMs) could help extend dental image assessment to settings with limited access to specialists. However, it remains unclear how well VLMs perform across dental imaging modalities and clinical tasks, or whether capable models can run within mobile hardware constraints. We introduce Pocket-Dentist, a large-scale curated dental imaging dataset, together with a comprehensive, deployment-aware benchmark for evaluating vision-language models across dental imaging modalities and clinical tasks. It includes over 50,800 images from more than 5,900 patients, spanning panoramic, periapical, and cephalometric radiographs and intraoral photographs, and covers four task types and 11 sub-tasks. We study dental-task performance across 13 representative VLMs under zero-shot and few-shot prompting, and examine the effects of fine-tuning the open-weight models. Our results show that, for zero-shot and few-shot settings, no VLM performs significantly better than others across all tasks. However, after fine-tuning, compact 4B VLMs such as Qwen3.5-4B become competitive with larger VLMs. We also observe that, even trained on large scale medical data, state-of-the-art medical VLMs provides no consistent advantage on dental understanding domain. To support practical deployment, we further proposed an on-device framework and deployed best performing VLMs. Our benchmark suggests that compact VLMs is practical for on-device use in dental image undertstanding. Our project page is available at https://anonymous.4open.science/r/Pocket-Dentist-Benchmark-Anonymous-62B3.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, medical AI, dental imaging, benchmark, on-device inference, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_azoHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a unified evaluation pipeline and a dental image benchmark for LLMs. This benchmark is applied to small models that can be deployed on-device to reach underserved communities.

Critical strengths

The motivation of the paper is strong: reaching underserved communities for a category of care that so many people need is a clear and commendable goal.

Involving dentists in the process is a good step toward designing and evaluating LLMs in a way that is more grounded in how medicine is practiced.

Critical weaknesses

The contribution is shallow:

  • The first contribution is not really one since the data already exists and they do not provide any justification or motivation as to why their data processing makes the benchmark more relevant to the goal of on-device dental care.
  • The “deployment aware evaluation framework” has nothing specific or groundbreaking. They simply measure latency of the models (which is a property of the models independent of the medical or dental field).
  • The third finding could be interesting but is underdeveloped. We see numbers that are unusual (larger models perform worse) but no explanation or insight into why. The part of the analysis with real dentists is also mostly relegated to a short section at the very end. The analysis is also suspicious since they seem to take a subset of the data that is known to be good before sending it to the dentists (they use BERTScore, which requires knowing the ground truth, to select the top 1/3 of the data, of which the dentists say 27% is usable in clinical settings. This could be as low as 9% of the whole data).

The literature seems a bit outdated. Similarly, the choice of models to benchmark seems a bit random: we have Qwen 3.5 4B compared to Qwen 2.5 7B when there is a Qwen 3.5 9B that would be a better and fairer comparison.

The main question of the paper mentions explainability but makes no contribution in this sense.

It would be interesting to also benchmark more API models. The premise of the study is that people who do not have access to a dentist have an iPhone 17 Pro (retails for $1,100). For an application that people would use essentially once to get diagnosed, a single API call could be more economically sound.

Questions for authors

Could you explain why the larger models perform worse than the smaller ones?

Reviewer_NfkrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines seven dental datasets into a VLM benchmark, compares prompting and fine-tuning strategies, and measures smartphone inference efficiency.

Critical strengths

The unified evaluation across multiple dental imaging modalities is useful. Comparing compact and larger models alongside deployment costs provides practical engineering insights.

Critical weaknesses

Clinical relevance remains insufficiently established. DR and DentalCaries omit available localization annotations and evaluate coarse image-level labels, without demonstrating their connection to clinical decisions. The claim that higher VQA accuracy improves triage is unsupported by triage evaluation. Moreover, only 14/51 assessed captions from the highest BERTScore tertile were fully correct and usable without amendment (Section 5.5). Cross-institution generalization is untested, and mobile experiments measure efficiency only. These limitations undermine the benchmark’s clinical motivation and support rejection.

Questions for authors
  1. What specific clinical decisions should these tasks support, and what evidence connects benchmark scores to those decisions?
  2. How does Pocket-Dentist differ from existing publicly available dental benchmarks in task design and clinical relevance, and what additional value does it provide beyond dataset aggregation?
Reviewer_3QqaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to investigate the capabilities of VLMs in oral-health-related tasks. It proposes Pocket-Dentist, a multimodal dental benchmark comprising seven data sources and four task types. It also constructs an evaluation framework to assess the zero-shot performance, few-shot performance as well as LoRA fine-tuning performance on diverse base models. Three models are deployed by an iOS application on an iPhone to evaluate the runtime performance, highlighting the practicability on VLM in solving the oral-health-related problems while running on mobile devices. The main contributions of the paper mainly lie in the comprehensive benchmark and the extensive comparison between models.

Critical strengths

This paper comprehensively compares across closed-source models and open-source models in different sizes.

Critical weaknesses
  1. The experiment results lack investigations of the underlying reasons. The results only report the metrics and rank of tested models. More analyses, especially CoT analysis, can be carried out to find out reasons behind the models’ judgements. Moreover, there is neither comparison between zero-shot and few-shot performance not comparison between LoRA fine-tuning and zero-shot performance, making the experiment incomplete.
  2. Some parts in the paper are incomplete or not rigorous. Section 5.5 does not involve the expert evaluation details. The repository in Appendix A.5.1 is empty with only a README.md in it. There are no explanations for the results shown in Figure 4 about the training dynamics of different models. In Section 3.3, tt and ntn_t (possibly the task number and the sample counts in the task) are not explained, pθp_\theta is not explained either and it represents the probability which differs from pip_i, the exemplar set Ek\mathcal{E}_k should be related with the task number tt. There are also typos in the abstract (Line 029 provides → provide, Line 031 is → are).
Questions for authors
  1. Given that CoT is a common technique to assess the model’s thinking process, what is the experiment results of the model’s CoT? Compared with zero-shot performance, 1-shot performance does not show significant gain, can you explain why?
  2. The on-device deployment uses iPhone 17 Pro, but it is much expensive than Android devices with equal or more memories. What are the results for on-device deployment on more widely-used Android devices?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.