- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CAPE combines a Claude Opus 4.8 controller with 13 deterministic vision tools to extract clinical charts into typed records. Its workflow classifies the chart, surveys its layout, extracts text and plotted values, and verifies measurements and record completeness. The authors also introduce 950 synthetic figures spanning 19 chart types, four difficulty tiers, and four rendering engines, with targets restricted to readable or derivable quantities. CAPE achieves Overall 0.780 versus 0.680 for the same-backbone Agent SDK baseline, winning on 17 chart types. Numeric is 0.662 versus 0.667. On real adverse-event severity charts, Overall reaches 0.508 versus the best baseline's 0.487.
Critical strengths
-
A useful benchmark formulation for clinical extraction. Section 3.3 on page 5 explicitly excludes latent generator quantities that cannot be recovered from the image. Appendix A specifies clinical chart coverage and controlled rendering variation. This design addresses a meaningful evaluation gap and could support systematic study of extraction beyond chart question answering.
-
Broad gains in schema coverage and labels with the backbone held fixed. Table 1 on page 8 improves Structure from 0.650 to 0.867 and Labels from 0.727 to 0.851 over the Agent SDK baseline. Table 8 on page 20 shows Overall improvements on 17 of 19 types, including multi-panel charts from 0.531 to 0.911. These results support the practical value of the specialized pipeline for assembling more complete outputs under the stated metric.
Critical weaknesses
-
The evaluation does not yet establish faithful recovery on real clinical figures. Table 2 and its discussion on pages 8-9 omit the real set's size, sources, and ground-truth construction and validation. The reported evaluation covers adverse-event severity charts, with Numeric only 0.219. Of Overall 0.508, 0.300 comes from Structure, which Appendix C defines as top-level field presence. This limits what the composite score establishes about usable numerical recovery. Appendix C on page 19 also does not explain whether Hungarian matching preserves time-value, series, and panel associations. If scalar arrays are matched independently, incorrect associations could receive credit. I cannot determine whether the implementation has this problem. These gaps prevent a reliable assessment of quantitative fidelity and generalization.
-
The added value of specialized measurement and verification remains insufficiently isolated. Section 4.1 on page 6 combines direct transcription of printed numbers with tool-based measurement, but Table 1 does not separate Numeric for these two sources. The Structure gain could also partly reflect the deterministic completeness gate. Giving the Agent SDK baseline the same gate would clarify the additional value of the remaining pipeline. Table 3 on page 9 supports a CV-tool benefit: removing the tools lowers Numeric from 0.675 to 0.639. However, reliability signals remain enabled in every configuration, Reflexion is not isolated, and the 76-chart ablation reports no uncertainty. Additionally, the round-trip check in Appendix B.3 on page 16 uses the same calibration that generated the values, so a consistently incorrect scale or offset can pass. Independent landmark checks may help, but their detection and correction performance is not quantified.
Questions for authors
-
Real-data validation: How many figures and source studies contribute to Table 2, and how were reference values obtained and checked? What uncertainty surrounds the reported differences between methods?
-
Schema selection and scoring: Section 5.2 states that all methods receive a target schema. Is it selected from the predicted or reference chart type? Does Hungarian matching preserve complete associated records? Please provide a scored curve example showing how incorrect time-value, series, or panel associations are penalized.
-
Attribution of the gains: Can you report Numeric separately for explicitly annotated values and values requiring geometric measurement, using the same field subsets for every method? How much of the Structure gain remains when the Agent SDK baseline receives the same deterministic completeness gate?
The authors did not respond and address the concerns. I will keep original rating.