- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PlotGround constructs numerical questions by linking real-world scientific figures to source tables and computing reference answers from those tables. It contains 1,119 human-verified questions from 1,066 bioRxiv preprints. Evaluation of 16 VLMs results in a 11–24% accuracy decline and error tolerance decreases from 5% to 2%. The biggest problems are reading from logarithmic axis and models snapping to tick marks instead of precise intermediate values. A separate coding-agent experiment compares figures, tables, and figures+tables where it finds that tabular data is more accurate and cheaper.
- I increased score from 2 to 3
Critical strengths
- Links real scientific figures to source data, allowing numerical answers to be reconstructed. All 1,119 questions are human-verified
- Evaluates 16 VLMs and reports accuracy/cost tradeoffs. The precision drop at tighter tolerances, logarithmic-axis errors, and tick-mark snapping are useful findings. The figure-versus-table experiment also shows the practical benefit of providing source data
Critical weaknesses
- Error analysis is limited to the chart-type breakdown. Results by axis scale, panel count, and visual density, ... would help explain where models fail
- Chart-to-table systems are not considered (to bridge the gap between charts and better performing tables). Also human eval would clarify how reliably the figures can be read within the 2% and 5% tolerances
Questions for authors
- How do plot-digitization or chart-to-table systems perform on this benchmark (at 2% and 5% tolerances)?
- How accurately can human readers answer a subset without access to source tables?
- Which cases account for the improvement from table-only to figure+table?
3 comments
Thanks for your additions of chart-type breakdown as well as including the performance of chart to table systems such as DePlot (plus the human evaluation) which from my point of view is very valuable.
Now chart to table systems dont seem to perform particularly well (not even for simple chart types such as Bar/Line chart). Do you have a reason why such systems have so much trouble? Is it the style thats different to traditional charts in computer science or the high complexity even for these simple chart types? Overall this would actually be a very positive finding because this directly validates the approach of using VLMs
For the human evaluation, where can I find this information in the paper? Whats the agreement between human annotators? To me it would also make sense if the authors could put in the most important/relevant results (e.g., chart2table comparison, human eval, ...) in the main paper instead of appendix as there is still a whole page left.
Lastly, are you planning to release the datasets? How much of it is redistributable (e.g., license information)?
If the authors adress these remaining questions, I would consider raising my score
Thank you very much for your continued engagement and for these thoughtful follow-up questions. We’re glad you found the additional analyses valuable.
Now chart to table systems dont seem to perform particularly well (not even for simple chart types such as Bar/Line chart). Do you have a reason why such systems have so much trouble? Is it the style thats different to traditional charts in computer science or the high complexity even for these simple chart types? Overall this would actually be a very positive finding because this directly validates the approach of using VLMs
Thank you for your question. Even for bar and line charts, many failures occur during table generation. Across the bar and line questions, the generated table omitted the requested value in 54.8% of cases for DePlot and 61.5% for UniChart.
To test whether chart complexity could explain the low accuracy, we evaluated progressively simpler subsets of the bar and line questions. We first restricted the analysis to panels containing a single chart, then to questions that did not require matching a series to a legend entry, and finally to those with category labels shorter than 20 characters. Accuracy moved from 25.8% to 28.3% for DePlot and stayed at 23.0% for UniChart. If complexity were the main cause, performance should recover on the simplest panels, and it did not. Both DePlot and UniChart were trained on a mixture of programmatically generated charts and charts collected from public websites. Differences in plotting style between these sources and research papers may contribute to the low accuracy we observe.
We agree this supports the case for evaluating VLMs directly on plot digitization. However, we would note that DePlot and UniChart are 282M and 202M parameter models from 2022 and 2023, so differences in model size and release date may also contribute to the gap.
For the human evaluation, where can I find this information in the paper? Whats the agreement between human annotators? To me it would also make sense if the authors could put in the most important/relevant results (e.g., chart2table comparison, human eval, ...) in the main paper instead of appendix as there is still a whole page left.
Thank you for your suggestion. Following your guidance, we have expanded Section 4.3 to report both the human evaluation and the comparison with the chart-to-table systems DePlot and UniChart. To assess agreement, we compared the independent reader’s readings with the original verification readings for the same 100 questions. The two readings agreed on whether the requested value was read within ±2% of the reference value in 80% of cases. Both readings were within tolerance on 78 questions, and neither was within tolerance on 2 others. On the remaining 20 questions, only one of the two readings was within tolerance.
Lastly, are you planning to release the datasets? How much of it is redistributable (e.g., license information)?
Thank you for your question. The full dataset is available for review at https://huggingface.co/datasets/anonymous10003/anonymous-repo. We will include a link to the public release of PlotGround-1k in the final version of the paper. We provide all 1,119 questions in a single Parquet file, with one row per question. Each row includes the corresponding source mapping and executable reconstruction. The questions, answers, and recipes are licensed under CC BY-SA 4.0. The copyright of the figures and source files belongs to the original authors.
For 840 questions, we include the figure images and the source data released by the authors. The remaining 279 questions come from preprints posted under an "all rights reserved" license, so we do not redistribute the corresponding figures or supplementary files. For these questions, image is set to None, and figure_file and source_file specify the expected local paths. We provide download_missing.py to retrieve the missing files from bioRxiv’s text and data mining bucket and render the figures using the same procedure we used for the bundled figures.
Thank you very much for your thoughtful feedback! We have added several detailed explanations and new experiments to address each of your comments. Please let us know if you have any further questions.
Thank you for the suggestion. Following your guidance, we report model accuracy at the ±5% tolerance by axis scale, panel count, and visual density in Appendix B.
Axis scale. We classified each question's target panel as linear or logarithmic from its tick spacing. Of the 1,119 questions, 1,077 fall on linear axes and 42 on log axes. All 16 models were more accurate on linear axes than on logarithmic axes, with gaps ranging from 14.9 to 41.4 percentage points. For GPT-5.6 Sol, the best model overall, accuracy was 88.8% on linear axes and 54.8% on logarithmic axes.
Panel count. We grouped questions by the number of panels in the source figure (1–3 / 4–5 / 6–8 / 9+, n = 256 / 331 / 344 / 188). Every model was less accurate on figures with nine or more panels than on figures with one to three, by 7.3 to 31.5 points.
Visual density. We approximate visual density by PNG bytes per megapixel of the image supplied to the models, taking compressed size as a proxy for how much local variation a figure contains. Across all 16 models, accuracy showed a consistent drop of 11.7–22.7 percentage points from the lowest to the highest visual density quartile.
We hope these comparisons help identify where models struggle to read values accurately from scientific figures.
Thank you for your question. We evaluated two published chart-to-table models, DePlot [1] and UniChart [2], as additional baselines, and have added them to Appendix B. Both models emit a single flat table and cannot express which panel a row belongs to. We therefore gave each model a crop of the panel identified in each question. We then used GPT-6.1 Sol to read the requested value out of the generated table, following how the DePlot paper pairs the model with a language model. The reader received only the question and the table. If it could not find the requested value, it could respond “not present in the table”, which we scored as incorrect.
† For these chart types, the generated table may omit the individual point or summary statistic needed to answer the question. Low accuracy can reflect both omissions and numerical reading errors. Other includes bubble plots, MA plots, strip plots, and survival plots.
Thank you for your question. We asked one independent reader to answer a subset of 100 questions without access to the source tables or reference answers. The reader used WebPlotDigitizer to extract the requested values and was evaluated on the same images as the models. The reader answered 93 of 100 questions correctly at the ±5% tolerance and 83 of 100 at the stricter ±2% tolerance. For comparison, GPT-5.6 Sol, the best model in Table 2, achieved 87.5% and 73.8% respectively on the full benchmark.
Thank you for your question. We would like to clarify that the small increase from 97.4% to 98.2% is consistent with run-to-run variation. At the ±5% tolerance, only six of 500 questions changed outcome (five gains and one loss). All five gains came from locating the correct entry in the supplementary data, and the single loss resulted from a unit conversion. The agent did not open the figure in any of these six cases.
[1] Liu, Fangyu, et al. "DePlot: One-shot visual language reasoning by plot-to-table translation." Findings of the Association for Computational Linguistics: ACL 2023. 2023.
[2] Masry, Ahmed, et al. "Unichart: A universal vision-language pretrained model for chart comprehension and reasoning." Proceedings of the 2023 conference on empirical methods in natural language processing. 2023.