acceptodds
Withdrawn from ICLR 2027

ChartDensity-Bench: Benchmarking MLLMs for Numerical Data Reconstruction under Visual Density

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) offer a promising approach for recovering numerical data from scientific charts, but their ability to reconstruct chart data from visually dense figures remains poorly understood. Existing chart understanding benchmarks primarily evaluate question answering or chart-level reasoning and provide limited support for evaluating structured numerical reconstruction from scientific figures. We introduce ChartDensity-Bench, a benchmark for evaluating MLLMs on structured numerical data reconstruction from compound chart figures under controlled visual density. Built from charts paired with source-level ground-truth data, ChartDensity-Bench systematically varies the number of simultaneously presented charts (), enabling controlled evaluation of density-induced degradation. We further propose a multi-dimensional evaluation framework covering structural reliability, reconstruction completeness, parseability, and numerical fidelity. Experiments on five recent MLLMs show that numerical reconstruction generally degrades as visual density increases, while the magnitude of degradation varies substantially across models. Chart-level paired comparisons further show that the same source chart can incur higher reconstruction error when embedded in denser visual contexts. These findings highlight visual density as an important and previously underexplored factor in MLLM chart data reconstruction and provide a systematic benchmark for evaluating model robustness in this setting.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Chart Data Reconstruction, Scientific Charts, Multimodal Document Understanding, Visual Density, Chart Understanding, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ognZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ChartDensity-Bench, a benchmark for reconstructing chart tables from compound scientific figures. It evaluates models on figures containing 1, 3, 6, or 9 charts and measures structural recovery, completeness, parseability, and numerical accuracy. The main finding is that reconstruction usually becomes worse as more charts are shown together.

Critical strengths

1.The paired setup evaluates the same source chart under different visual contexts, making the density effect easier to study than in ordinary benchmark comparisons.

2.The evaluation separates chart-level failures, missing points, and numerical errors, which gives a more useful diagnosis than a single chart-to-table score.

Critical weaknesses

1.The practical significance of the task is unclear. Chart-to-table extraction is well studied, and compound figures can often be cropped into individual charts before extraction. Without comparing against a crop-then-extract pipeline, it is unclear whether the reported degradation reflects an important limitation or an avoidable preprocessing issue.

2.The density effect is not cleanly isolated. Increasing the number of charts also changes each chart’s effective resolution and may trigger different image preprocessing across models. In addition, numerical errors are computed only on successfully matched points, which may make the results sensitive to the matching procedure.

Questions for authors

1.Why should a model process the full compound figure instead of first detecting and cropping individual subplots? How much does the crop-then-extract baseline improve performance?

2.Does the density effect remain when per-chart resolution and overall input size are controlled? How are missing, duplicated, or unmatched points penalized in the numerical metrics?

Reviewer_UFt4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper benchmarks open-source and proprietary LLMs for their reasoning capabilities on single-grid (1×3) and multi-grid (1×3, 2×3, 3×3) bar and line charts by combining ICPR CHART-Infographics charts into multi-grid layouts. The authors propose different evaluation criteria to assess four dimensions, such as structural reliability, parseability, completeness, and numerical fidelity.

Critical strengths
  • The shortcomings of the current current VLMs on the multi-chart setup is an interesting topic
  • Different levels of VLM capabilities to asses (starting from number of multi charts) is interesting
Critical weaknesses
  • The contributions of the paper are very limited; with its current form it would be better considered as white paper as evaluation of recent VLMs than a top ML conference paper.
  • The paper is very hard to read, hardly comprehensible and missing details.
  • No related work section.
Questions for authors
  • What is the rationale of limiting to only 1x3, 2x3, 3x3 grids, what is the rationale of limiting to bar and line charts?
  • How are the charts selected to combine?
  • I see no related work section. There are other multi-chart datasets available with more variety of chart types, and different grid combinations (e.g. Kondic, Jovana, et al. "Chartnet: A million-scale, high-quality multimodal dataset for robust chart understanding. CVPR (2026).). Could you please add a related work section?
Reviewer_ZRweOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ChartDensity-Bench to evaluate numerical data reconstruction by multimodal large language models under increasing chart density, but its central claims are undermined by serious methodological flaws, questionable evaluation procedures, and factual inconsistencies.

Critical strengths
  • The paper addresses an underexplored and meaningful problem by enabling knowledge graphs to proactively reorganize and self-evolve via cross-chunk hidden relation discovery rather than remaining entirely static.
Critical weaknesses
  1. Confounded Experimental Design: Increasing the number of subplots simultaneously changes image composition, input processing demands, and output burden, making it impossible to isolate visual density as the actual cause of performance degradation.

  2. Flawed Numerical Normalization: MAE and RMSE are normalized using the ground-truth range of successfully matched points, meaning that changes in point coverage can artificially alter the reported numerical errors even when predictions on shared points remain unchanged.

  3. Redundant Structural Metrics: Under the paper's stated evaluation procedure, RFV and CAR are mathematically equivalent, calling into question the claimed independence of these evaluation dimensions.

  4. Invalid Completeness Measurement: The proposed ACR metric is based on the ratio of extracted to ground-truth entries without an explicit upper bound or sufficient correctness constraints, potentially rewarding duplicated or excessive predictions.

  5. Inconsistent Matching Rules: The evaluation protocol leaves numerical x-value matching tolerances unspecified and provides contradictory requirements for categorical x-values, raising concerns about the validity of point alignment.

  6. Selection Bias in Numerical Evaluation: Numerical errors are computed only on successfully aligned and parseable charts, systematically excluding failed reconstructions and potentially biasing both the reported errors and subsequent statistical analyses.

  7. Insufficient Statistical Evidence: The mixed-effects analysis does not establish a statistically significant difference in nine-chart density sensitivity between Kimi and Gemini, weakening claims of Kimi's superior robustness over all evaluated models.

  8. Unsupported Mechanistic Interpretation: Although the manuscript promises attention-level analysis and invokes attention dilution to explain its findings, it provides no direct analysis of attention distributions or internal model representations.

  9. Factual and Citation Errors: The manuscript incorrectly cites the 2026 RAVE paper as a 2025 publication and references earlier model families rather than clearly documenting the exact evaluated model versions.

  10. Unexplained Experimental Anomalies: Qwen's RFV unexpectedly increases from 62.07% at six subplots to 97.40% at nine subplots, a striking anomaly that receives insufficient investigation or explanation.

The accumulation of factual inaccuracies, contradictory evaluation definitions, questionable methodological choices, and unsupported interpretations raises serious concerns about the scientific reliability of this work.

More importantly, I suspect that this manuscript may have been produced through an automated auto-research pipeline without sufficient human verification, although the observed issues alone cannot establish its actual authorship process.

Questions for authors

Can the authors genuinely guarantee the factual correctness and scientific integrity of this work, including the provenance of the dataset, the validity of the evaluation metrics, the accuracy of the references, and the authenticity and reproducibility of the experimental results, and provide concrete, independently verifiable evidence supporting these claims?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.