acceptodds
Withdrawn from ICLR 2027

CAPE: A Clinical Chart Agent for Precision Vision Extraction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) can answer questions about charts yet struggle to extract their quantitative content precisely and comprehensively. The difficulty is particularly pronounced in clinical charts, where specialized measurements appear within dense visual layouts. Accurate recovery of this information is essential for further quantitative analysis and evidence synthesis. To close this gap, we introduce CAPE, an MLLM-driven agent harness that combines semantic reasoning with pixel-level measurement for clinical chart extraction. The MLLM identifies the quantities to extract and locates the relevant chart elements. Dedicated vision tools follow this guidance to measure the plotted values and return reliability signals that inform remeasurement within a staged workflow. Record-level verification checks the completeness and organization of the extraction before the workflow concludes. To evaluate extraction quality, we develop a novel synthetic data generation approach that produces clinical charts paired with ground truth restricted to quantities readable or derivable from each figure. Using this approach, we construct a benchmark of 950 figures spanning 19 clinical chart types. CAPE raises the Overall extraction score on this benchmark from 0.68 to 0.78 over the strongest baseline built on the Claude Agent SDK and achieves higher scores on 17 of the 19 chart types, producing more complete and correctly organized records. CAPE also achieves the highest Overall score among all evaluated methods on a real clinical chart dataset.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Multimodal Large Language Models, Clinical Chart Extraction, AI Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bLVkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAPE combines a Claude Opus 4.8 controller with 13 deterministic vision tools to extract clinical charts into typed records. Its workflow classifies the chart, surveys its layout, extracts text and plotted values, and verifies measurements and record completeness. The authors also introduce 950 synthetic figures spanning 19 chart types, four difficulty tiers, and four rendering engines, with targets restricted to readable or derivable quantities. CAPE achieves Overall 0.780 versus 0.680 for the same-backbone Agent SDK baseline, winning on 17 chart types. Numeric is 0.662 versus 0.667. On real adverse-event severity charts, Overall reaches 0.508 versus the best baseline's 0.487.

Critical strengths
  1. A useful benchmark formulation for clinical extraction. Section 3.3 on page 5 explicitly excludes latent generator quantities that cannot be recovered from the image. Appendix A specifies clinical chart coverage and controlled rendering variation. This design addresses a meaningful evaluation gap and could support systematic study of extraction beyond chart question answering.

  2. Broad gains in schema coverage and labels with the backbone held fixed. Table 1 on page 8 improves Structure from 0.650 to 0.867 and Labels from 0.727 to 0.851 over the Agent SDK baseline. Table 8 on page 20 shows Overall improvements on 17 of 19 types, including multi-panel charts from 0.531 to 0.911. These results support the practical value of the specialized pipeline for assembling more complete outputs under the stated metric.

Critical weaknesses
  1. The evaluation does not yet establish faithful recovery on real clinical figures. Table 2 and its discussion on pages 8-9 omit the real set's size, sources, and ground-truth construction and validation. The reported evaluation covers adverse-event severity charts, with Numeric only 0.219. Of Overall 0.508, 0.300 comes from Structure, which Appendix C defines as top-level field presence. This limits what the composite score establishes about usable numerical recovery. Appendix C on page 19 also does not explain whether Hungarian matching preserves time-value, series, and panel associations. If scalar arrays are matched independently, incorrect associations could receive credit. I cannot determine whether the implementation has this problem. These gaps prevent a reliable assessment of quantitative fidelity and generalization.

  2. The added value of specialized measurement and verification remains insufficiently isolated. Section 4.1 on page 6 combines direct transcription of printed numbers with tool-based measurement, but Table 1 does not separate Numeric for these two sources. The Structure gain could also partly reflect the deterministic completeness gate. Giving the Agent SDK baseline the same gate would clarify the additional value of the remaining pipeline. Table 3 on page 9 supports a CV-tool benefit: removing the tools lowers Numeric from 0.675 to 0.639. However, reliability signals remain enabled in every configuration, Reflexion is not isolated, and the 76-chart ablation reports no uncertainty. Additionally, the round-trip check in Appendix B.3 on page 16 uses the same calibration that generated the values, so a consistently incorrect scale or offset can pass. Independent landmark checks may help, but their detection and correction performance is not quantified.

Questions for authors
  1. Real-data validation: How many figures and source studies contribute to Table 2, and how were reference values obtained and checked? What uncertainty surrounds the reported differences between methods?

  2. Schema selection and scoring: Section 5.2 states that all methods receive a target schema. Is it selected from the predicted or reference chart type? Does Hungarian matching preserve complete associated records? Please provide a scored curve example showing how incorrect time-value, series, or panel associations are penalized.

  3. Attribution of the gains: Can you report Numeric separately for explicitly annotated values and values requiring geometric measurement, using the same field subsets for every method? How much of the Structure gain remains when the Agent SDK baseline receives the same deterministic completeness gate?

Reviewer_cF7oOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to extract accurate and complete data from clinical charts, where dense layouts and specialized measurements make the task difficult for multimodal models.

CAPE combines an MLLM with thirteen visual measurement tools. A staged workflow organizes the output fields, reads the chart, and checks the extracted record.

On 950 synthetic figures covering nineteen chart types, CAPE raises Overall from 0.680 to 0.780 over the Agent SDK baseline. It also has the highest reported Overall score on real adverse-event charts.

Critical strengths
  1. I like the benchmark design. Real clinical figures often come without their underlying data, which makes extraction hard to evaluate. Section 3 and Appendix A describe generators for nineteen chart types, with four difficulty levels and several rendering engines. The ground truth includes only values that can be read or derived from the figure. This is a sensible way to avoid testing information that only the generator knows.

  2. The system is easy to follow: the model decides what to read, and the tools measure the relevant pixels. Sections 4.2-4.3 and Figure 3 make this division concrete. Table 8 shows higher Overall scores on seventeen of nineteen chart types, with the largest gain on multi-panel figures. That suggests the workflow is useful for organizing complex extraction tasks.

Critical weaknesses
  1. The staged workflow uses a StateFlow-style design, and the record review draws on Reflexion, as the paper itself notes. Combining model guidance, visual tools, and feedback is also familiar from systems such as PixelCraft. The aim-and-measure interface and two levels of verification could be useful additions, but the paper needs to explain what is technically new about them. As written, I see a useful system for clinical charts, but the case for methodological novelty is weak.

  2. The comparison with existing methods is also too limited. The experiments use general-purpose MLLMs and a coding agent, but leave out the specialized chart-extraction and visual-agent methods discussed in the related work. I would like to see a comparison with a closer system such as PixelCraft, adapted to the extraction task, and with methods such as ChartOCR or LineFormer on chart types they support. These methods need not cover all nineteen types; comparisons on shared subsets with the same target fields and scoring would be informative.

  3. On the synthetic benchmark, the overall gain does not show an improvement in numerical extraction over the strongest baseline. In Table 1, Overall rises by 0.100, but Numeric is 0.662 versus 0.667 for Agent SDK. The authors acknowledge that most of the gain comes from Structure and Labels. Structure, however, only checks whether top-level fields are present (Appendix C). I would like to see whether the specialized tools improve numerical extraction when both systems use the same workflow, schema checks, backbone, and resource budget.

  4. The ablations do not tell me how much the proposed feedback mechanism helps. Measurement reliability signals are kept in every setting, and Overview and Reflexion are removed together with staged control in one row. I would like separate ablations for the signals and verification steps, plus repeated runs showing how often remeasurement actually fixes an error.

Questions for authors
  1. What is technically new in the aim-and-measure interface and the two levels of verification compared with the closest existing agent designs?

  2. How does CAPE compare with specialized visual agents and chart-extraction methods on shared chart types, using the same target fields and scoring protocol?

  3. With the workflow, schema checks, backbone, and budget held fixed, do specialized measurement tools improve Numeric over general-purpose tools on the main benchmark? Which quantities and chart types benefit?

  4. What happens when measurement reliability signals alone are turned off? How often does remeasurement fix a wrong value? Please report variation across repeated runs and the extra model calls, tokens, and latency used by verification.

Reviewer_Xwe1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CAPE, an agent framework for extracting structured information from clinical charts. The method combines multimodal language models with dedicated computer vision tools to identify chart elements, measure numerical values, and verify the extracted results. The authors also introduce a synthetic benchmark containing 950 charts across 19 clinical chart types. Experiments show that CAPE improves the overall extraction score compared with several multimodal models and a general coding agent.

Critical strengths

1.The paper addresses a practical and meaningful problem. Extracting structured numerical information from clinical charts is challenging, and the proposed benchmark covers a diverse range of chart types with different levels of visual complexity. The effort to construct ground truth based on information that can actually be recovered from the figures is particularly valuable.

  1. The proposed framework is intuitive and reasonably well designed. Combining semantic understanding from multimodal models with dedicated computer vision tools is a promising approach to chart extraction. The staged workflow and verification mechanism also help produce more complete and organized results. The overall performance improvements across most chart types demonstrate the potential usefulness of this approach.
Critical weaknesses
  1. My main concern is that the improvements in numerical extraction are less convincing than the overall results suggest. In Table 1, CAPE achieves a higher overall score than the strongest agent baseline, but its Numeric score is slightly lower, at 0.662 compared with 0.667. Most of the improvement comes from Structure and Labels. Furthermore, the Structure metric only measures the presence of top level fields rather than the correctness of their contents. This raises questions about how well the overall metric reflects the actual quality of extracted clinical data. The use of unordered numerical matching may also overlook incorrect associations between values and their corresponding time points or treatment groups.

  2. The evaluation on real clinical charts is relatively limited. The real chart experiments focus on adverse event severity plots, and the dataset size and annotation procedure are not sufficiently described. Although CAPE achieves the highest Numeric score, its value is only 0.219, compared with 0.206 for the strongest individual model. Without additional information about the reliability and variability of these results, it is difficult to determine whether the improvement is meaningful or whether the approach generalizes well to other real clinical chart types.

Questions for authors
  1. Could the authors clarify whether the numerical matching procedure preserves the correspondence between extracted values and their associated time points or treatment groups? Would the main performance conclusions remain the same under an evaluation that explicitly checks these associations?

  2. Could the authors provide more information about the number of real clinical charts, their sources, and the ground truth annotation procedure? In particular, are the improvements in numerical accuracy consistent across individual figures, and are they statistically reliable?

1 comment

The authors did not respond and address the concerns. I will keep original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.