- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a unified evaluation pipeline and a dental image benchmark for LLMs. This benchmark is applied to small models that can be deployed on-device to reach underserved communities.
Critical strengths
The motivation of the paper is strong: reaching underserved communities for a category of care that so many people need is a clear and commendable goal.
Involving dentists in the process is a good step toward designing and evaluating LLMs in a way that is more grounded in how medicine is practiced.
Critical weaknesses
The contribution is shallow:
- The first contribution is not really one since the data already exists and they do not provide any justification or motivation as to why their data processing makes the benchmark more relevant to the goal of on-device dental care.
- The “deployment aware evaluation framework” has nothing specific or groundbreaking. They simply measure latency of the models (which is a property of the models independent of the medical or dental field).
- The third finding could be interesting but is underdeveloped. We see numbers that are unusual (larger models perform worse) but no explanation or insight into why. The part of the analysis with real dentists is also mostly relegated to a short section at the very end. The analysis is also suspicious since they seem to take a subset of the data that is known to be good before sending it to the dentists (they use BERTScore, which requires knowing the ground truth, to select the top 1/3 of the data, of which the dentists say 27% is usable in clinical settings. This could be as low as 9% of the whole data).
The literature seems a bit outdated. Similarly, the choice of models to benchmark seems a bit random: we have Qwen 3.5 4B compared to Qwen 2.5 7B when there is a Qwen 3.5 9B that would be a better and fairer comparison.
The main question of the paper mentions explainability but makes no contribution in this sense.
It would be interesting to also benchmark more API models. The premise of the study is that people who do not have access to a dentist have an iPhone 17 Pro (retails for $1,100). For an application that people would use essentially once to get diagnosed, a single API call could be more economically sound.
Questions for authors
Could you explain why the larger models perform worse than the smaller ones?