acceptodds
Under review as a conference paper at ICLR 2027

PlotGround: Grounding Plot Digitization in Real Scientific Figures and Their Source Data

Abstract

Scientific figures often encode quantitative results that are not readily available in machine-readable form, making accurate plot digitization important for verifying and reusing published findings. Yet it remains unclear how accurately current models recover plotted values from real scientific figures, as existing benchmarks rely largely on synthetic charts or cover only a limited range of chart types. We introduce PlotGround, an automated pipeline for building plot digitization benchmarks from real scientific figures and their author-released source data. PlotGround maps figures to source tables, identifies reconstructable panels, and generates quantitative questions with source-grounded reference values. We use PlotGround to construct PlotGround-1k, a human-verified benchmark of 1,119 questions from 1,066 bioRxiv preprints. Across sixteen multimodal models, the best reaches 87.5% accuracy at a % relative-error tolerance. Tightening the tolerance to % lowers every model's accuracy by 11-24 percentage points, revealing a gap between approximate visual reading and precise quantitative recovery. PlotGround's paired figure-source structure lets us compare how accurately the same values are recovered from figures and from source tables. Providing source tables instead of figures raises a coding agent's accuracy from 90.0% to 97.4% while cutting cost by 72%.

Primary area:
datasets and benchmarks
Keywords:
Plot Digitization, Benchmark, Scientific Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GBuVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PlotGround constructs numerical questions by linking real-world scientific figures to source tables and computing reference answers from those tables. It contains 1,119 human-verified questions from 1,066 bioRxiv preprints. Evaluation of 16 VLMs results in a 11–24% accuracy decline and error tolerance decreases from 5% to 2%. The biggest problems are reading from logarithmic axis and models snapping to tick marks instead of precise intermediate values. A separate coding-agent experiment compares figures, tables, and figures+tables where it finds that tabular data is more accurate and cheaper.

  • I increased score from 2 to 3
Critical strengths
  • Links real scientific figures to source data, allowing numerical answers to be reconstructed. All 1,119 questions are human-verified
  • Evaluates 16 VLMs and reports accuracy/cost tradeoffs. The precision drop at tighter tolerances, logarithmic-axis errors, and tick-mark snapping are useful findings. The figure-versus-table experiment also shows the practical benefit of providing source data
Critical weaknesses
  • Error analysis is limited to the chart-type breakdown. Results by axis scale, panel count, and visual density, ... would help explain where models fail
  • Chart-to-table systems are not considered (to bridge the gap between charts and better performing tables). Also human eval would clarify how reliably the figures can be read within the 2% and 5% tolerances
Questions for authors
  • How do plot-digitization or chart-to-table systems perform on this benchmark (at 2% and 5% tolerances)?
  • How accurately can human readers answer a subset without access to source tables?
  • Which cases account for the improvement from table-only to figure+table?
3 comments

Thank you very much for your thoughtful feedback! We have added several detailed explanations and new experiments to address each of your comments. Please let us know if you have any further questions.

Error analysis is limited to the chart-type breakdown. Results by axis scale, panel count, and visual density, ... would help explain where models fail

Thank you for the suggestion. Following your guidance, we report model accuracy at the ±5% tolerance by axis scale, panel count, and visual density in Appendix B.

Axis scale. We classified each question's target panel as linear or logarithmic from its tick spacing. Of the 1,119 questions, 1,077 fall on linear axes and 42 on log axes. All 16 models were more accurate on linear axes than on logarithmic axes, with gaps ranging from 14.9 to 41.4 percentage points. For GPT-5.6 Sol, the best model overall, accuracy was 88.8% on linear axes and 54.8% on logarithmic axes.

Panel count. We grouped questions by the number of panels in the source figure (1–3 / 4–5 / 6–8 / 9+, n = 256 / 331 / 344 / 188). Every model was less accurate on figures with nine or more panels than on figures with one to three, by 7.3 to 31.5 points.

Visual density. We approximate visual density by PNG bytes per megapixel of the image supplied to the models, taking compressed size as a proxy for how much local variation a figure contains. Across all 16 models, accuracy showed a consistent drop of 11.7–22.7 percentage points from the lowest to the highest visual density quartile.

We hope these comparisons help identify where models struggle to read values accurately from scientific figures.

How do plot-digitization or chart-to-table systems perform on this benchmark (at 2% and 5% tolerances)?

Thank you for your question. We evaluated two published chart-to-table models, DePlot [1] and UniChart [2], as additional baselines, and have added them to Appendix B. Both models emit a single flat table and cannot express which panel a row belongs to. We therefore gave each model a crop of the panel identified in each question. We then used GPT-6.1 Sol to read the requested value out of the generated table, following how the DePlot paper pairs the model with a language model. The reader received only the question and the table. If it could not find the requested value, it could respond “not present in the table”, which we scored as incorrect.

Chart typeDePlot (±5%)DePlot (±2%)UniChart (±5%)UniChart (±2%)
Overall13.011.110.99.2
Bar27.624.325.221.4
Volcano†0.00.00.00.0
Scatter†3.43.42.31.1
Box†8.93.61.81.8
Line16.714.311.99.5
Violin†5.95.95.95.9
Other†0.00.00.00.0

† For these chart types, the generated table may omit the individual point or summary statistic needed to answer the question. Low accuracy can reflect both omissions and numerical reading errors. Other includes bubble plots, MA plots, strip plots, and survival plots.

How accurately can human readers answer a subset without access to source tables?

Thank you for your question. We asked one independent reader to answer a subset of 100 questions without access to the source tables or reference answers. The reader used WebPlotDigitizer to extract the requested values and was evaluated on the same images as the models. The reader answered 93 of 100 questions correctly at the ±5% tolerance and 83 of 100 at the stricter ±2% tolerance. For comparison, GPT-5.6 Sol, the best model in Table 2, achieved 87.5% and 73.8% respectively on the full benchmark.

Which cases account for the improvement from table-only to figure+table?

Thank you for your question. We would like to clarify that the small increase from 97.4% to 98.2% is consistent with run-to-run variation. At the ±5% tolerance, only six of 500 questions changed outcome (five gains and one loss). All five gains came from locating the correct entry in the supplementary data, and the single loss resulted from a unit conversion. The agent did not open the figure in any of these six cases.

[1] Liu, Fangyu, et al. "DePlot: One-shot visual language reasoning by plot-to-table translation." Findings of the Association for Computational Linguistics: ACL 2023. 2023.

[2] Masry, Ahmed, et al. "Unichart: A universal vision-language pretrained model for chart comprehension and reasoning." Proceedings of the 2023 conference on empirical methods in natural language processing. 2023.

Thanks for your additions of chart-type breakdown as well as including the performance of chart to table systems such as DePlot (plus the human evaluation) which from my point of view is very valuable.

Now chart to table systems dont seem to perform particularly well (not even for simple chart types such as Bar/Line chart). Do you have a reason why such systems have so much trouble? Is it the style thats different to traditional charts in computer science or the high complexity even for these simple chart types? Overall this would actually be a very positive finding because this directly validates the approach of using VLMs

For the human evaluation, where can I find this information in the paper? Whats the agreement between human annotators? To me it would also make sense if the authors could put in the most important/relevant results (e.g., chart2table comparison, human eval, ...) in the main paper instead of appendix as there is still a whole page left.

Lastly, are you planning to release the datasets? How much of it is redistributable (e.g., license information)?

If the authors adress these remaining questions, I would consider raising my score

Thank you very much for your continued engagement and for these thoughtful follow-up questions. We’re glad you found the additional analyses valuable.

Now chart to table systems dont seem to perform particularly well (not even for simple chart types such as Bar/Line chart). Do you have a reason why such systems have so much trouble? Is it the style thats different to traditional charts in computer science or the high complexity even for these simple chart types? Overall this would actually be a very positive finding because this directly validates the approach of using VLMs

Thank you for your question. Even for bar and line charts, many failures occur during table generation. Across the bar and line questions, the generated table omitted the requested value in 54.8% of cases for DePlot and 61.5% for UniChart.

To test whether chart complexity could explain the low accuracy, we evaluated progressively simpler subsets of the bar and line questions. We first restricted the analysis to panels containing a single chart, then to questions that did not require matching a series to a legend entry, and finally to those with category labels shorter than 20 characters. Accuracy moved from 25.8% to 28.3% for DePlot and stayed at 23.0% for UniChart. If complexity were the main cause, performance should recover on the simplest panels, and it did not. Both DePlot and UniChart were trained on a mixture of programmatically generated charts and charts collected from public websites. Differences in plotting style between these sources and research papers may contribute to the low accuracy we observe.

We agree this supports the case for evaluating VLMs directly on plot digitization. However, we would note that DePlot and UniChart are 282M and 202M parameter models from 2022 and 2023, so differences in model size and release date may also contribute to the gap.

For the human evaluation, where can I find this information in the paper? Whats the agreement between human annotators? To me it would also make sense if the authors could put in the most important/relevant results (e.g., chart2table comparison, human eval, ...) in the main paper instead of appendix as there is still a whole page left.

Thank you for your suggestion. Following your guidance, we have expanded Section 4.3 to report both the human evaluation and the comparison with the chart-to-table systems DePlot and UniChart. To assess agreement, we compared the independent reader’s readings with the original verification readings for the same 100 questions. The two readings agreed on whether the requested value was read within ±2% of the reference value in 80% of cases. Both readings were within tolerance on 78 questions, and neither was within tolerance on 2 others. On the remaining 20 questions, only one of the two readings was within tolerance.

Lastly, are you planning to release the datasets? How much of it is redistributable (e.g., license information)?

Thank you for your question. The full dataset is available for review at https://huggingface.co/datasets/anonymous10003/anonymous-repo. We will include a link to the public release of PlotGround-1k in the final version of the paper. We provide all 1,119 questions in a single Parquet file, with one row per question. Each row includes the corresponding source mapping and executable reconstruction. The questions, answers, and recipes are licensed under CC BY-SA 4.0. The copyright of the figures and source files belongs to the original authors.

For 840 questions, we include the figure images and the source data released by the authors. The remaining 279 questions come from preprints posted under an "all rights reserved" license, so we do not redistribute the corresponding figures or supplementary files. For these questions, image is set to None, and figure_file and source_file specify the expected local paths. We provide download_missing.py to retrieve the missing files from bioRxiv’s text and data mining bucket and render the figures using the same procedure we used for the bundled figures.

Reviewer_f7vgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PlotGround constructs plot-digitization benchmarks by linking real scientific figures to author-deposited source tables. A three-stage coding-agent pipeline maps figures to tables, records reconstruction recipes for suitable panels, and generates questions about individual plotted values. Human verification yields PlotGround-1k, with 1,119 questions from 1,066 bioRxiv preprints across ten chart types. Table 2 reports results for sixteen multimodal models. GPT-5.6 Sol achieves 87.5% accuracy at a relative-error tolerance of ±5%, falling to 73.8% at ±2%. A separate experiment on 500 rewritten questions finds that giving a coding agent source tables improves accuracy and reduces cost relative to providing figures, with manuscript text available in both conditions. Eight unresolved figure–data discrepancies are excluded from the benchmark and discussed as exploratory findings.

Critical strengths
  1. A useful benchmark with traceable numerical references. Sections 3.2–3.4 specify explicit figure–table mappings, reconstruction operations, and rejection of ambiguous cases. Appendix A.3 on page 17 documents WebPlotDigitizer checks of all 1,205 candidates and adjudication of flagged cases, yielding 1,119 retained questions. These safeguards support the credibility of the labels and make the benchmark useful for evaluating numerical recovery from authentic, often multi-panel scientific figures.

  2. The paired experiment provides actionable evidence about access to source data. Table 3 on page 8 holds the 500 questions, agent, prompt, and manuscript context fixed while changing the available evidence. Replacing figures with tables raises accuracy from 90.0% to 97.4%, reduces mean cost from 0.408 to 0.113, and lowers latency from 139 to 51 seconds. This supports using accessible source tables for quantitative recovery in the studied setting.

Critical weaknesses
  1. The strict precision results need calibration against the final human audit and input resolution. Appendix A.3 on page 17 describes adjudication of 131 flagged candidates, with answers either confirmed or corrected before retention. It does not report the final distribution of discrepancies between WebPlotDigitizer readings and reference values, or the numbers retained unchanged versus corrected. Section 4.2 caps full figures at 1,600 pixels on the longest edge, but whether reviewers used the same inputs is unspecified. Small panels and logarithmic axes can make the ±2% criterion sensitive to positional uncertainty. The accuracy drop in Table 2 is established, but the contributions of model imprecision, input resolution, and any residual figure–source disagreement remain unclear.

  2. The construction decisions are difficult to reproduce and assess from the supplied PDF. Appendix C on pages 19–20 supplies evaluation and question-rewriting prompts, but I could not locate the three construction-stage prompts, a benchmark access link, or a release plan. Section 3.4 also filters candidates using agreement between model readings and source values, and prioritizes clear targets, without quantifying exclusions from that check. These omissions limit independent verification of the reference values and assessment of the selected evaluation population. Access to reconstruction artifacts and exclusion statistics would substantially reduce this concern.

Questions for authors
  1. Calibration clarification: Can you report the final distribution of discrepancies between WebPlotDigitizer readings and reference values on the retained benchmark, together with the numbers of unchanged, corrected, and excluded cases among the 131 flagged candidates? Were the readings obtained from the same images used in Table 2? These details would help assess the interpretation of the ±2% results.

  2. Construction clarification: Are the benchmark, per-question source mappings and executable reconstructions, and the three construction-stage prompts available for review? Please also report how many candidates the synthesis-stage reading check excluded and, if audited, whether exclusions reflected invalid mappings or valid but difficult targets. This would clarify both reproducibility and the scope of the reported accuracies.

1 comment

We thank Reviewer f7vg for their positive comments and helpful feedback on our work.

Calibration clarification: Can you report the final distribution of discrepancies between WebPlotDigitizer readings and reference values on the retained benchmark, together with the numbers of unchanged, corrected, and excluded cases among the 131 flagged candidates? Were the readings obtained from the same images used in Table 2? These details would help assess the interpretation of the ±2% results.

Thank you for the suggestion. We have expanded Appendix A.3 to report the details of the human audit: Among the 131 flagged candidates, 40 were confirmed to be correct and kept unchanged, 5 were retained after their reference values were corrected, and the remaining 86 were excluded. Human reviewers used the same images at the same resolution as the models in Table 2. On the retained benchmark, WebPlotDigitizer readings were within ±2% of the final reference values for 1,099 of the 1,119 questions (98.2%).

Construction clarification: Are the benchmark, per-question source mappings and executable reconstructions, and the three construction-stage prompts available for review? Please also report how many candidates the synthesis-stage reading check excluded and, if audited, whether exclusions reflected invalid mappings or valid but difficult targets. This would clarify both reproducibility and the scope of the reported accuracies.

Thank you for the question. The full dataset is available for review at https://huggingface.co/datasets/anonymous10003/anonymous-repo. We will include a link to the public release of PlotGround-1k in the final version of the paper. We provide all 1,119 questions in a single Parquet file, with one row per question. Each row includes the corresponding source mapping and executable reconstruction. We have also added all three construction-stage prompts to the appendix.

For 840 questions, we include the figure images and the source data released by the authors. The remaining 279 questions come from preprints posted under an “all rights reserved” license, so we do not redistribute the corresponding figures or supplementary files. For these questions, image is set to None, and figure_file and source_file specify the expected local paths. We provide download_missing.py to retrieve the missing files from bioRxiv’s text and data mining bucket and render the figures using the same procedure we used for the bundled figures.

The synthesis-stage reading check excluded 497 candidates. The agent traces attribute 379 exclusions to invalid source mappings. In the other 118 cases, the traces indicate that the mappings were valid but the targets were difficult to identify or read. Of these 118 targets, 88 lacked a visible identifier in the figure. The remaining 30 were too small to read, overlapped with other elements, or had ambiguous labels.

Reviewer_cmVdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PlotGround, a pipeline that uses real scientific figures and their source data to build a benchmark for reading numbers from plots. The benchmark contains 1119 questions from 1066 bioRxiv papers. The authors test 16 multimodal models to see how accurately they can read the values shown in these figures.

Critical strengths
  1. Grounding reference answers in author-released source data, together with human verification, provides a useful basis for assessing the numerical precision of models’ figure-reading capabilities.
  2. The benchmark contains over 1000 questions drawn from more than 1000 scientific papers, representing a substantial data collection and verification effort.
Critical weaknesses
  1. I am not fully convinced that this benchmark addresses an important practical need. When I read papers, the key numbers are often written directly on the figures, or I can understand the results just by looking at the plots. If I need more detailed numbers, I would usually look for the source data. So when would someone need a model to read such precise values from an image? A few concrete examples showing why this matters in actual research would help.
  2. Some of the findings are fairly expected. A stricter error threshold will give lower accuracy, and giving an agent the source data should make its job easier. The numbers are useful to know, but I would like to see what they mean in practice. For example, would the reading errors actually change the result of an analysis or the conclusion someone draws from a figure?
Questions for authors

See weaknesses.

2 comments

Thank you very much for your helpful feedback and support for the paper! We have carefully updated the paper to incorporate your suggestions.

I am not fully convinced that this benchmark addresses an important practical need. When I read papers, the key numbers are often written directly on the figures, or I can understand the results just by looking at the plots. If I need more detailed numbers, I would usually look for the source data. So when would someone need a model to read such precise values from an image? A few concrete examples showing why this matters in actual research would help.

Thank you for the comment. We agree that source data are the first place to look when precise values are needed, but they can be difficult to obtain. For example, Gabelica et al. [1] requested data for 1,792 papers whose authors had indicated a willingness to share them. Only 123 authors provided the requested data.

When source data are unavailable, researchers can extract numerical estimates from published figures. In a meta-analysis, researchers can use these estimates to combine or compare results across studies. Wang et al. [2] digitized survival curves from 26 immunotherapy trials to estimate the percentage of patients alive at 12 months in each group. They used these numbers to ask whether trials showing a larger treatment benefit at 12 months also showed a larger benefit over the full follow-up period. Similarly, Noble et al. [3] synthesized 175 studies to quantify how incubation temperature affects reptile development. A reader can see from a plot that eggs hatch sooner at higher temperatures. To quantify that effect across studies, however, researchers need numerical values such as the mean incubation time and its variability in each temperature group. Pick et al. [4] reported that 42% of the papers in this meta-analysis presented at least some of the required data in figures. We have added the reptile development example to the Introduction to illustrate why researchers may need numerical values from figures to quantify effects across studies.

Some of the findings are fairly expected. A stricter error threshold will give lower accuracy, and giving an agent the source data should make its job easier. The numbers are useful to know, but I would like to see what they mean in practice. For example, would the reading errors actually change the result of an analysis or the conclusion someone draws from a figure?

Thank you for this suggestion. Reading errors can affect the outcome of an analysis, and prior work shows this for both human and model readers. Turner et al. [5] analyzed 43 interrupted time series using both the original data and values independently digitized by four researchers. The estimates generally agreed. For one researcher’s digitized values, however, the significance tests did not always agree with those on the original data. In 1 of 43 series, the two analyses disagreed on whether the outcome shifted immediately after the intervention (level change). In 2 of 43 series, they disagreed on whether the trend changed after the intervention (slope change).

For multimodal models, MatSciBench [6] attributed 68.9% of the 781 substantive errors analyzed on materials science questions with images to visual data extraction. We have added the Turner et al. example to Section 4.3 to illustrate how digitization errors can change the outcome of a statistical test.

[1] Gabelica, Mirko, Ružica Bojčić, and Livia Puljak. "Many researchers were not compliant with their published data sharing statement: a mixed-methods study." Journal of clinical epidemiology 150 (2022): 33-41.

[2] Wang, Zi-Xian, et al. "Correlation of milestone restricted mean survival time ratio with overall survival hazard ratio in randomized clinical trials of immune checkpoint inhibitors: a systematic review and meta-analysis." JAMA Network Open 2.5 (2019): e193433.

[3] Noble, Daniel WA, Vaughn Stenhouse, and Lisa E. Schwanz. "Developmental temperatures and phenotypic plasticity in reptiles: A systematic review and meta‐analysis." Biological Reviews 93.1 (2018): 72-97.

[4] Pick, Joel L., Shinichi Nakagawa, and Daniel WA Noble. "Reproducible, flexible and high‐throughput data extraction from primary literature: the metaDigitise R package." Methods in Ecology and Evolution 10.3 (2019): 426-431.

[5] Turner, Simon Lee, et al. "Effect estimates can be accurately calculated with data digitally extracted from interrupted time series graphs." Research Synthesis Methods 14.4 (2023): 622-638.

[6] Zhang, Junkai, et al. "MatSciBench: Benchmarking the reasoning ability of large language models in materials science." Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026.

The authors’ response cleared up my concern about when this benchmark would be useful in practice. They also shared some evidence that errors in reading values from plots can affect analysis results. It would be even better to see this demonstrated on PlotGround itself, for example through experiments showing how those errors can change the scientific conclusions drawn from a figure. Overall, I think this paper is a useful contribution to the community, so I’m raising my rating to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.