- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents COCAgent for diagnosing strabismus subtypes and generating reports from nine-gaze photographs. It uses ocular measurements to build a graph, selects evidence paths with SHAP scores, and checks candidate diagnoses against clinical rules. The authors also introduce StrabBench, a dataset of 1,065 patients with subtype labels, clinician-reviewed reports, and evidence annotations. On a 104-patient test set, the system reports 92.0% weighted F1 and 93.3% accuracy.
Critical strengths
- The clinical problem is worth studying. Showing the gaze-position findings behind a prediction could help clinicians check the result.
- StrabBench provides reports and position-level annotations, not just subtype labels. These could help assess whether a model picks up the right findings, even when its final diagnosis is correct.
Critical weaknesses
1. Several inconsistencies in the evaluation and annotations make the results hard to trust.
Supplementary Sec. 2.3.1 states that a wrong diagnosis gets zero credit at all downstream levels. Averaged over the same cases, these scores therefore cannot exceed diagnosis accuracy. Yet in Supplementary Table 2, InternVL3.5-8B has 6.7% accuracy but 28.5% Position Alignment and 38.4% Treatment BERTScore. Qwen3.5-4B also has 17.3% accuracy but 20.4% Position Alignment. These numbers do not fit the stated protocol.
Main-paper Table 4 has another counting issue. With one chain per case over 104 cases, mean chain lengths of 1.31, 1.85, and 1.19 imply 136, 192, and 124 total edges. None can give the reported Cov@10 values of 92.5%, 80.8%, and 94.1% under the supplement’s pooled-edge formula, even after rounding. E.g., for 124 edges, the neighboring possible values are 116/124 = 93.5% and 117/124 = 94.4%. Position Alignment also changes definition: the main paper requires exact Position/Position Group matches, while the supplement expands groups into individual positions before computing coverage.
There are clear mismatches in the benchmark examples too. In Supplementary Fig. 3, Example 4 is labeled Exotropia and described as Non-A/V, but its graph root says A-pattern Exotropia. Example 6 is labeled and described as Vertical, yet its graph root says V-pattern Esotropia. The same report calls Position 2 “downgaze,” contradicting both the fixed mapping and its own later text. Some reference reports also infer numerical upgaze–downgaze differences and clinical thresholds from photographs, although the supplied rules explicitly say uncalibrated images cannot establish these thresholds.
2. The inference procedure and ablations do not line up, and the causal role of the chain remains unproven.
Sec. 4.3 says the final label is the candidate that passes verification. COC is used afterward to generate the report. Why, then, does adding COC in Table 3(c)→(d) change accuracy from 90.4% to 89.4%? The supplementary verification prompt also asks for a New_Label, but its role in the main pipeline is unclear. Without knowing how the final label is actually chosen, it is hard to interpret these ablations.
The gains do not clearly come from the chain either. Rules and verification alone reach 90.4% accuracy. Adding COC lowers accuracy and drops AccAng from 98.1% to 93.3%. The full system reaches 93.3% accuracy, only three more correct cases overall than the no-chain version, with Evidence Matching also added. A comparison against non-causal evidence selection or memory under the same setup is missing, so the benefit of causal structure is not isolated.
There is also a mismatch in what counts as a chain. Eq. 6 defines one continuous path to one target, but Supplementary Fig. 1 calls a path to Esotropia plus a separate edge to V-pattern a six-link chain. How the two three-class DAGs are combined into six-class candidates and a joint explanation is not explained clearly. Evidence Matching is described as choosing between two chains, yet the results say it “removes exactly those links.” Where does this edge removal happen? Also, an edge outside the ten most frequent CMU edges may still occur in the population statistics; that alone does not make it clinically unsupported.
The more basic issue is whether these relationships are causal at all. The paper interprets discovered feature dependencies as direct causes without adequately stating the assumptions or independently checking the directions. SHAP explains contributions to a classifier’s output, not clinical causation. Checking paths against frequencies produced by the same pipeline does not establish causality either. The COC prompt simply tells the model that each variable causes the next and asks it to explain the links. That may yield a plausible account, but it does not test whether those links are true.
3.The labels are described as being assigned from nine-gaze photographs, while the supplied rules state that these images cannot establish the quantitative thresholds for A/V patterns. Were the subtype labels confirmed using independent clinical measurements, or do they represent image-based judgments of a compatible trend? This distinction matters for what the classification results actually establish. The paper also gives little quantitative evidence of annotation reliability beyond clinician review, which is particularly concerning given the conflicting examples. Finally, Gemini drafted the reference reports and is also the backbone of COCAgent, but there is no independent clinical assessment to separate factual report quality from similarity to the reference style.
4.Qwen3.5-9B and Hulu-Med-7B both get 2.9% accuracy, while InternVL3.5-8B gets 6.7%. The larger models also do much worse than their smaller counterparts. This is possible, but the paper gives little explanation. Were the image inputs and model-specific formats checked? How were label variants, refusals, and unparseable answers scored? Without these checks or examples of failed outputs, it is hard to tell whether the low scores reflect diagnostic failures or problems with the evaluation setup.
Questions for authors
Pls see weaknesses above.