- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Overjet-1, a dental vision-language model that processes multiple images and modalities within a patient examination. The authors construct a Dental Graph linking tooth identities, clinical findings, and geometry across views, and use it to generate 1.63 million supervised records covering 122 tasks, including annotation-derived reasoning traces. The model is trained through supervised fine-tuning and Dental Consistency Reinforcement Learning (DC-RL), which combines task correctness with grounding, evidence, format, and tooth-layout rewards. Evaluation includes the external MMOral-OPG benchmark, the patient-disjoint internal Overjet-Bench, external COde anomaly classification, specialist comparisons, and ablations on two backbones. The authors report 71.5% closed-answer accuracy and a 75.1% open-answer judge score on MMOral, and a 78.8 macro score on Overjet-Bench. Additional modality-removal and shuffling experiments assess input utilization, while non-dental experiments examine the broader usefulness of consistency rewards.
Critical strengths
S1: A useful case-level formulation and substantial grounded supervision. The Dental Graph explicitly links tooth identities, clinical findings, and geometry across views and modalities, allowing the same model to address localization, numbering, question answering, reporting, and cross-input tasks. This targets a practically important limitation of evaluating dental images independently, and the construction of 122 task types is a substantial contribution.
S2: A meaningful empirical package with relevant controls. The paper includes external MMOral and COde evaluations, patient-disjoint internal evaluation, SFT and DC-RL ablations, a second backbone, and case-level confidence intervals. Table 11 additionally evaluates modality removal and patient-mismatched inputs on two 200-case slices. These results provide substantive evidence of post-training gains and of dependence on the supplied modalities, beyond the qualitative examples.
Critical weaknesses
W1: The clinical validity of the main internal benchmark is not sufficiently established. Much training supervision comes from production systems, and Appendix A acknowledges clinically plausible annotation errors. Overjet-Bench uses the same clinical database, but Appendix F.1 does not make clear whether its reference boxes and clinical facts were independently adjudicated rather than inherited from source systems. Dentist verification is explicitly described for the displayed case studies (Appendix J), which does not establish benchmark-wide reliability. Moreover, Appendix G.3 assigns zero to parsing failures, and reports are assessed with ROUGE-L/token F1; no cross-model parsing audit or blinded agreement with clinical judgments is reported. Thus, the 45.3-point macro advantage may combine clinical improvement with agreement to annotation and output conventions. MMOral and COde support external transfer, but do not directly resolve these concerns for cross-view grounding and reporting. A bounded audit of existing benchmark references and predictions would make this central result more convincing.
W2: The reward specification makes the claimed isolated effect of spatial consistency difficult to interpret. For at least two predicted tooth boxes, Equation (6) defines the tooth-box term as 0.9 i + 0.1 s, whereas Appendix D.3's no-spatial control uses i for the corresponding term. The contrast therefore changes the absolute-grounding coefficient as well as removing consistency, despite statements that absolute grounding remains unchanged (Appendix G.4). Table 3's 3.4-point gain supports the full reward design, but not unambiguously the addition of s at fixed weights. Also, Appendix D.3 describes a sum of geometry components, while Algorithm 3 selects them through mutually exclusive branches. The handling of a target containing both tooth and lesion geometry, including how the grounding score remains bounded, needs clarification. Please identify the actual implementation and reconcile the formulae and the interpretation of the ablation.
Questions for authors
Q1: How were Overjet-Bench reference boxes and clinical facts obtained, and which were independently annotated or adjudicated by dentists rather than derived from the source systems? Please distinguish benchmark-wide validation from the verified examples in Appendix J. Can existing audit results, or a small blinded subset of the current test records and model predictions, quantify reference errors and agreement between the automatic scorer and clinical correctness? A per-model parsing-failure breakdown would also help assess whether the ranking is sensitive to output conventions.
Q2: What exact geometry aggregation and no-spatial reward were used in the reported runs? In particular, does the control retain the 0.9 coefficient on absolute tooth-box grounding, and are tooth boxes, finding boxes, and keypoints scored jointly or through exclusive branches? If the implementation differs from the written definition, please correct it. If the coefficient changes, please provide any existing weight-matched evidence or qualify the conclusion as a comparison between reward designs rather than an isolated addition of spatial consistency.
5 comments
Response to Reviewer 9K7J — Part 2/2
Q1, Part 4. Parsing failures and ranking sensitivity
A per-model parsing-failure breakdown would also help assess whether the ranking is sensitive to output conventions.
A per-model parsing-failure breakdown
The system prompt used in our evaluation specifies a structured output format; however, models may occasionally fail to comply despite using a standardized answer schema and parser. We therefore quantified, for each model, how often outputs failed to parse, separately from parseable but incorrect answers (Table 16). Across the 12 evaluated models, the average first-pass parsing-failure rate was 1.18%. MedVLM-R1 had the highest rate at 3.17% (95/3,000), compared with 0.13% (4/3,000) for Overjet-1. To assess the effect on the results, we reran failed outputs with up to three additional generations using the same input, system prompt, and parser for each model. After retries, every model’s Overall score changed by at most 1.03 points, and the top four models retained their rankings. Overjet-1’s parsing failures decreased to zero, its score changed from 73.53 to 73.66, and it remained first. Thus, recovering parsing failures leaves the main conclusion unchanged.
Q2. Geometry aggregation and the no-spatial control
What exact geometry aggregation and no-spatial reward were used in the reported runs? In particular, does the control retain the 0.9 coefficient on absolute tooth-box grounding, and are tooth boxes, finding boxes, and keypoints scored jointly or through exclusive branches? If the implementation differs from the written definition, please correct it. If the coefficient changes, please provide any existing weight-matched evidence or qualify the conclusion as a comparison between reward designs rather than an isolated addition of spatial consistency.
We thank the reviewer for the close reading of the reward definition and apologize for the inconsistent descriptions. In our experiments, the no-spatial control retained the 0.9 coefficient on absolute tooth-box grounding: for predictions with at least two valid tooth boxes, the full reward uses 0.9i + 0.1s, while the control uses 0.9i. The single-box behavior and the other reward terms remain unchanged, and the topology-only contribution for positive targets without reference geometry is disabled in the control. We have corrected the no-spatial expression in Appendix D.3 and aligned it with Appendix G.4 so that the fixed-coefficient comparison is explicit.
Your comments have helped us clarify the benchmark validation and reward specification. We would be happy to address any remaining questions. If our responses and additional results resolve your concerns, we would be grateful if you could consider a higher score.
Post-Rebuttal Assessment
Thank you for the detailed response and additional analyses. The clarification regarding independent expert annotation substantially addresses my concern about reference-label provenance. The parsing-failure analysis also alleviates my concern about output-format effects on the ranking. I regard the fixed-coefficient no-spatial control as clarified and withdraw the corresponding concern.
My remaining concern is the validation of clinical text-output evaluation. Expert verification of reference annotations and QA validity strengthens the benchmark, but does not establish how well ROUGE-L and token F1 reflect the clinical correctness of model-generated reports and syntheses. The response explains these metrics but does not provide the requested agreement analysis or a blinded clinical audit of model predictions.
The sample proportion of text-output questions also does not directly establish their contribution to Overall. Under the original nine-family macro average, the reporting family contributes one ninth regardless of its sample count. Please clarify the revised aggregation weights. A small clinical-scoring audit would help establish the practical impact of this remaining concern.
The response has improved my assessment and substantially narrowed my concerns. I nevertheless retain my Weak Reject recommendation at present because the validation supporting the clinical text-quality claims remains incomplete.
Response to Reviewer 9K7J — Part 1/2
Thank you very much for the detailed review. We greatly appreciate that you recognized the case-level formulation of the Dental Graph, the scale of the grounded supervision across 122 task types, and the empirical package of external evaluations, patient-disjoint evaluation, ablations on two backbones and case-level confidence intervals. Your questions concern the reliability of the benchmark and the exact reward specification, and we have addressed both in the revision. See below:
Q1, Part 1. Benchmark annotation
We first note that we have revised and substantially extended Overjet-Bench in the revised manuscript to thoroughly evaluate the efficacy of the model on multi-image and multimodal tasks. The benchmark now includes 3,000 QA pairs across single-image, multi-image and multimodal questions.
The ground truth, including reference boxes and clinical facts for teeth, anatomy, restorations, lesions and periodontal status, was constructed by a team of 31 expert annotators: 17 specialists and 14 general dentists. The test set was labeled from scratch, with no pre-filling from the source systems. Each case was labeled by one annotator and then checked by one reviewer, so every case was reviewed by at least two experts.
From the labeled clinical facts, QA pairs were constructed using the same generation step as the training QA pairs. The validity of the QA pairs was checked manually by a team of 31 annotators. The final 3,000 QA pairs were selected from the validated pairs.
Q1, Part 2. Benchmark validation and illustrative examples
The overjet-bench annotation/adjudication is separate from the verification of the illustrative examples in Appendix J. Each QA pair correctness was validated by the team of expert annotators, whereas Appendix J lists examples of model predictions.
Q1, Part 3. Scoring agreement and clinical correctness
Agreement between scoring metrics and clinical correctness
We distinguish evaluation criteria by question type. Multiple-choice answers are evaluated by matching the selected option to the reference. Grounding is evaluated through task-specific box matching at IoU ≥ 0.5, with tooth-identity and source-view requirements where applicable. These metrics quantify agreement with the annotated targets (Appendix F.1).
For whole-image and per-tooth reports, we use ROUGE-L; open clinical synthesis uses token F1. These metrics measure content overlap with the dentist-annotated and checked references. Because the outputs mostly describe structured dental information, overlap provides a useful signal of agreement with the reference content. The clinical correctness is more directly evaluated by fact based questions (multiple-choice, grounding, and keyword extraction, etc.). The report-generation family also only accounts for 135 of the 3,000 benchmark examples (4.5%), including 90 evaluated with ROUGE-L (3.0%) and 45 with token F1 (1.5%) (Table 12).