acceptodds
Under review as a conference paper at ICLR 2027

CaptureChart: How Reliable Is Chart Understanding across Capture Conditions?

Abstract

Reading structured visual information from photographed charts matters for automated field inspections, factory monitoring, and mobile question answering. As vision–language models advance, evaluation benchmarks must evolve to assess their growing capabilities and remaining limitations. Yet existing real-photo benchmarks are difficult to scale because adding new charts or capture conditions requires costly new photography. Natural captures also combine multiple degradations, making their individual effects on chart understanding difficult to isolate. Hence, we introduce CaptureChart, a fully digital benchmark for studying chart understanding under simulated capture. Starting from 1,000 CharXiv charts, a Blender-based pipeline models the paper surface, surrounding scene, lighting, and camera imaging to produce 32,000 aligned views. Each chart has an electronic original, a clean capture, and 30 variations across four capture families and 11 mechanisms at selected intensity levels. To examine difficult reasoning questions under capture degradation, we add 1,151 visual-evidence questions linked to 173 reasoning questions. Each probe targets a chart fact used in a checked solution to its parent question, allowing us to test whether models recover the evidence needed for the final answer. Across 14 vision–language models and 30 capture variations, mean accuracy falls relative to clean captures for descriptive, reasoning, and visual-evidence questions; reasoning accuracy drops by 9.4 percentage points. Reasoning is often more sensitive than descriptive reading. Linked evidence answers can change correctness even when aggregate accuracy changes little, and specific capture mechanisms expose distinct model weaknesses. Overall, CaptureChart offers a scalable way to study chart understanding under controlled, simulated capture conditions. The benchmark is available at https://anonymous-hf.com/a/7if3j4z0hjpy/.

Primary area:
datasets and benchmarks
Keywords:
Chart Question Answering, Vision-Language Models, Robustness Evaluation, Simulated Image Capture, Visual Evidence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TXZtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CaptureChart, a benchmark for evaluating chart understanding under simulated capture conditions. A Blender-based pipeline transforms 1,000 CharXiv charts into 32,000 views covering surface deformation, lighting, framing, and camera effects while retaining the original questions and answers. It additionally constructs 1,151 visual-evidence questions linked to 173 reasoning questions to examine consistency between parent answers and supporting facts. Evaluations across 14 vision–language models reveal accuracy degradation, differences in sensitivity across tasks and capture conditions, and correctness changes obscured by aggregate accuracy. The benchmark provides controlled comparisons for studying chart understanding under visual degradation.

Critical strengths
  1. The rendering pipeline supports controlled, paired comparisons across capture conditions while preserving the underlying charts and questions. This reusable design makes it easier to investigate individual capture factors and extend evaluation to additional chart collections.
  2. The evaluation covers 14 models and 30 capture variations, with detailed analyses across question categories. Reporting both correct-to-wrong and wrong-to-correct transitions provides useful information about answer stability that aggregate accuracy alone can conceal.
Critical weaknesses
  1. The simulated conditions lack validation. The paper specifies rendering parameters for low light, overexposure, and reflections, but does not establish how the chosen levels relate to actual photographs. It also remains unclear whether people can still answer the questions under severe degradation. Some failures may therefore reflect missing information rather than poor model robustness.

  2. The benefit over existing perturbation benchmarks is unclear. CHAOS [1] already studies chart QA under synthetic perturbations. CaptureChart reports accuracy drops, model-specific sensitivities, and correctness flips. Its examples of missed ticks and unreadable axis labels are useful, but could also arise under ordinary resizing or blur. What additional finding requires the proposed capture simulation?

  3. VE answers do not directly reveal how the parent question was solved. Asking for a named series at a specified position tells the model where to look, whereas the parent question may require finding that evidence independently. Correct VE answers therefore do not establish correct perception during parent answering. Conversely, a correct parent answer may use another solution path. Requiring every probe to be correct also makes the comparison sensitive to probe count.

[1] Chen, Yu-Fan et al. “CHAOS: Chart Analysis with Outlier Samples.” ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2026): n. pag.

Questions for authors
  1. How were the capture severity levels selected? Have you checked whether they resemble actual photographs and whether the retained questions remain answerable? A small comparison with real captures and human answers would help clarify what the reported failures measure.

  2. Which specific finding goes beyond what CHAOS or simpler image perturbations can reveal? For example, does comparing your rendered captures with ordinary blur or resizing on the same charts expose different errors?

  3. What happens if the model receives verified VE answers when solving the parent question? This would help distinguish errors that remain after the relevant facts are supplied from difficulties in locating those facts in the image.

Reviewer_mSsUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CaptureChart, which uses Blender to simulate photographed charts under different lighting, deformation and camera conditions. It evaluates 14 VLMs and adds visual-evidence questions to examine errors behind reasoning failures.

Critical strengths
  1. the problem is interesting and practically relevant.
  2. controlled captures allow comparison across conditions while keeping the chart and questions fixed. The evidence questions also provide useful diagnosis beyond final accuracy.
  3. the coverage on models studied is comprehansive
Critical weaknesses
  1. Since the capture process is synthetic and its parameters are known, some distortions could potentially be corrected (we could reverse engineer it with neural nets). There is no image-restoration baseline, including an oracle version using rendering metadata. This would help separate recoverable capture distortion from chart-understanding failures.

  2. There is no human performance reference under the degraded conditions as ceilings (or maybe some models are better than humans). Some questions may become unanswerable because blur or overexposure removes the required evidence.

  3. No real photographed charts are evaluated. It remains unclear whether the simulated conditions reproduce real capture difficulty or preserve model rankings on real photos.

Questions for authors

No question

Reviewer_X7VvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CaptureChart, a benchmark for evaluating chart understanding under simulated image-capture conditions. Using a Blender-based pipeline, the authors generate 32,000 views from 1,000 CharXiv charts, covering 11 degradation mechanisms. The benchmark also includes 1,151 visual-evidence questions linked to reasoning tasks. Experiments on 14 vision-language models reveal performance degradation, greater sensitivity of reasoning tasks in many settings, and answer instability under different capture conditions.

Critical strengths
  1. Practical significance and systematic benchmark design. The paper addresses an important limitation of existing chart benchmarks by evaluating visual understanding under controlled capture conditions. The configurable synthesis pipeline enables systematic comparisons across different degradation mechanisms while preserving the original questions and answers.

  2. Comprehensive evaluation and useful findings. The experiments cover 14 vision-language models and 30 capture variations. Beyond aggregate accuracy, the analysis of correctness transitions and linked visual-evidence questions provides useful insights into model robustness and failure patterns.

Critical weaknesses
  1. Limited validation of simulation realism. Although the benchmark is motivated by real-world chart photography, the generated capture conditions are not quantitatively validated against real photographs. It remains unclear how well the simulated artifacts and their severity levels represent practical capture conditions. This limits the conclusions that can be drawn about real-world robustness.

  2. Limited coverage and validation of visual-evidence questions. The 1,151 visual-evidence questions are derived from only 173 of the 1,000 reasoning questions. While the paper describes automated validation and selected manual checks, it does not provide a systematic human annotation-quality assessment or sufficient analysis of the selected subset's representativeness. These limitations affect the generality and reliability of the evidence-level diagnostic conclusions.

Questions for authors
  1. Can the authors provide a quantitative comparison between simulated captures and real photographs, or otherwise justify the realism and severity settings of the proposed capture conditions?

  2. What determines the selection of the 173 reasoning questions for visual-evidence decomposition? Can the authors provide a systematic human validation of the generated questions and analyze whether this subset is representative of the full reasoning benchmark?

Comments on the paper

1 comment
AuthorsOpenReview ↗

the general response for all reviewer

We thank the reviewers for their feedback. New evidence supports real-photo correspondence and the value of VE facts beyond question cues.

1. New experimental evidence

ExperimentKey resultsSupported claim
Correspondence with real photographsFeature similarity: our paper captures 0.967; CHAOS reconstruction 0.930; blur/down-sampling 0.943; ChartR-style perturbations 0.948.Our captures are closest to real photographs in the measured feature space among these methods.
Failure agreement: Cohen’s κ = 0.47 for our captures, versus 0.12, 0.22, and 0.11 for the same baselines.Our captures agree more closely with real photographs on which questions models fail.
VE-answer interventions in parent solvingReference answers vs. parent only: accuracy improves by 31.5–34.3 pp across three models. Qwen3.8-27B improves from 54.1% to 88.4%; its clean-to-perturbed gap shrinks from 13.5 to 1.2 pp.Providing reference facts improves parent solving and largely closes the capture-induced gap in this experiment.
Reference answers vs. questions only: Qwen3.8-27B improves from 57.6% to 88.4%.Factual answers contribute substantially beyond location cues in the VE questions.
Models’ own VE answers: all-correct sets repair 54.8–72.1% of parent errors; sets containing errors repair 8.7–14.7%.Correct intermediate answers are associated with greater recovery of parent errors.
Capture-image human QCThree reviewers judged an average of 145/155 pairs (93.5%) answerable overall; excluding strong defocus, 145/150 (96.7%).Most sampled pairs under the remaining conditions retain discernible information needed to answer.
VE-question human QCThree reviewers judged an average of 198/200 questions (99.0%) answerable on electronic originals, with reference answers and model responses hidden.The sampled VE questions generally request information discernible from the original charts.
Image restorationSource-assisted geometric rectification improves the 27B model by 3.4 pp (95% chart-bootstrap interval: 0.6–6.4), but does not improve the two smaller models. Generic restoration reduces all three models’ accuracy by about 5 pp.Across diverse capture degradations, simple rectification and restoration do not consistently improve accuracy; effective recovery may require pipelines tailored to photographed-chart QA.

Settings. The real-photo comparison uses 200 ChartQA charts and 598 corresponding WildDoc photographs, holding chart content, questions, and reference answers fixed across image versions. QA evaluation covers eight VLMs and 333 questions. Similarity uses frozen Qwen3.5-4B vision-encoder features; failure agreement considers questions each model answered correctly on electronic originals.

VE interventions use 173 CharXiv parents and three Qwen models (4B, 8B, and 27B), comparing no VE input, VE questions only, reference answers, and models’ own answers. These parents were selected from all 1,000 reasoning questions for decomposition into independently answerable chart facts. Results refer to the evaluated VE set and original 30 perturbation conditions. Capture QC samples five pairs per capture condition; VE QC uses electronic originals. Restoration uses 100 reasoning questions.

2. Two revisions

Defocus severity. All five sampled strong-defocus pairs were judged unanswerable, while sampled mild- and moderate-defocus pairs remained answerable. We have removed strong defocus from the main text while retaining the other two levels, preserving coverage of the defocus mechanism.

VE granularity. The median number of children per parent is 3, with excessive decomposition concentrated in a small number of parents. Large child sets make the all-correct criterion more stringent and affect analyses based on it; we have capped the number of children per parent to address these cases. The direct interventions above support VE’s intended role: testing access to supporting facts and their value for reasoning.

3. Practical value

We will release the data and reusable capture pipeline, which converts existing electronic chart datasets into capture-domain images while retaining their QA annotations. We have packaged the pipeline as a callable API and added screen artifacts, including moiré and display pixel-grid patterns. Its practical use also extends to training: we have generated SFT data for a 4B VLM to read product labels on second-hand clothing, bags, and shoes, obtaining better performance than training directly on real photographs in that application. The pipeline thus supports controlled evaluation and training beyond chart benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.