- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CaptureChart, a benchmark for evaluating chart understanding under simulated capture conditions. A Blender-based pipeline transforms 1,000 CharXiv charts into 32,000 views covering surface deformation, lighting, framing, and camera effects while retaining the original questions and answers. It additionally constructs 1,151 visual-evidence questions linked to 173 reasoning questions to examine consistency between parent answers and supporting facts. Evaluations across 14 vision–language models reveal accuracy degradation, differences in sensitivity across tasks and capture conditions, and correctness changes obscured by aggregate accuracy. The benchmark provides controlled comparisons for studying chart understanding under visual degradation.
Critical strengths
- The rendering pipeline supports controlled, paired comparisons across capture conditions while preserving the underlying charts and questions. This reusable design makes it easier to investigate individual capture factors and extend evaluation to additional chart collections.
- The evaluation covers 14 models and 30 capture variations, with detailed analyses across question categories. Reporting both correct-to-wrong and wrong-to-correct transitions provides useful information about answer stability that aggregate accuracy alone can conceal.
Critical weaknesses
-
The simulated conditions lack validation. The paper specifies rendering parameters for low light, overexposure, and reflections, but does not establish how the chosen levels relate to actual photographs. It also remains unclear whether people can still answer the questions under severe degradation. Some failures may therefore reflect missing information rather than poor model robustness.
-
The benefit over existing perturbation benchmarks is unclear. CHAOS [1] already studies chart QA under synthetic perturbations. CaptureChart reports accuracy drops, model-specific sensitivities, and correctness flips. Its examples of missed ticks and unreadable axis labels are useful, but could also arise under ordinary resizing or blur. What additional finding requires the proposed capture simulation?
-
VE answers do not directly reveal how the parent question was solved. Asking for a named series at a specified position tells the model where to look, whereas the parent question may require finding that evidence independently. Correct VE answers therefore do not establish correct perception during parent answering. Conversely, a correct parent answer may use another solution path. Requiring every probe to be correct also makes the comparison sensitive to probe count.
[1] Chen, Yu-Fan et al. “CHAOS: Chart Analysis with Outlier Samples.” ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2026): n. pag.
Questions for authors
-
How were the capture severity levels selected? Have you checked whether they resemble actual photographs and whether the retained questions remain answerable? A small comparison with real captures and human answers would help clarify what the reported failures measure.
-
Which specific finding goes beyond what CHAOS or simpler image perturbations can reveal? For example, does comparing your rendered captures with ordinary blur or resizing on the same charts expose different errors?
-
What happens if the model receives verified VE answers when solving the parent question? This would help distinguish errors that remain after the relevant facts are supplied from difficulties in locating those facts in the image.
the general response for all reviewer
We thank the reviewers for their feedback. New evidence supports real-photo correspondence and the value of VE facts beyond question cues.
1. New experimental evidence
Settings. The real-photo comparison uses 200 ChartQA charts and 598 corresponding WildDoc photographs, holding chart content, questions, and reference answers fixed across image versions. QA evaluation covers eight VLMs and 333 questions. Similarity uses frozen Qwen3.5-4B vision-encoder features; failure agreement considers questions each model answered correctly on electronic originals.
VE interventions use 173 CharXiv parents and three Qwen models (4B, 8B, and 27B), comparing no VE input, VE questions only, reference answers, and models’ own answers. These parents were selected from all 1,000 reasoning questions for decomposition into independently answerable chart facts. Results refer to the evaluated VE set and original 30 perturbation conditions. Capture QC samples five pairs per capture condition; VE QC uses electronic originals. Restoration uses 100 reasoning questions.
2. Two revisions
Defocus severity. All five sampled strong-defocus pairs were judged unanswerable, while sampled mild- and moderate-defocus pairs remained answerable. We have removed strong defocus from the main text while retaining the other two levels, preserving coverage of the defocus mechanism.
VE granularity. The median number of children per parent is 3, with excessive decomposition concentrated in a small number of parents. Large child sets make the all-correct criterion more stringent and affect analyses based on it; we have capped the number of children per parent to address these cases. The direct interventions above support VE’s intended role: testing access to supporting facts and their value for reasoning.
3. Practical value
We will release the data and reusable capture pipeline, which converts existing electronic chart datasets into capture-domain images while retaining their QA annotations. We have packaged the pipeline as a callable API and added screen artifacts, including moiré and display pixel-grid patterns. Its practical use also extends to training: we have generated SFT data for a 4B VLM to read product labels on second-hand clothing, bags, and shoes, obtaining better performance than training directly on real photographs in that application. The pipeline thus supports controlled evaluation and training beyond chart benchmarks.