- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Chart-SR proposes a data and training framework to improve evidence-sensitive chart reasoning in MLLMs. It constructs charts and controlled interventions and automatically verifiable annotations pairs. The model is trained with GRPO to identify visual changes, judge their effects, and update or preserve answers.
Critical strengths
-
Important research problem. The paper addresses the gap between answer correctness and faithful visual grounding. This is important for charts containing dense numerical evidence.
-
Simple and verifiable training framework.
Critical weaknesses
1. Limited methodological novelty: Counterfactual reasoning and evidence-grounded RL have already been explored by DeFacto and CFPO, while ChartPoint, Chart-FR1, and VisDoT address fine-grained chart grounding. Chart-SR appears to mainly introduce another task-specific data construction and reasoning format, with limited methodological significance.
2. Insufficient evidence for improved visual perception: The method may primarily teach a structured ground-and-solve pattern rather than improve visual perception. The rewards do not directly verify extracted values or question-relevant evidence. Zero-/few-shot in-context learning with the same output format and direct grounding evaluations would help establish the necessity of RL training, and verify whether it’s just this responding structure helps improve the performance.
3. Missing and outdated baselines and benchmarks: confused by Table 2 omitting the original Qwen3-VL backbones, and Table 3 includes a 4B comparison on different benchmarks. why? moreover, it relies heavily on older MLLMs, including GPT-4o despite GPT-5's release in August 2025 (The submission deadline of this paper is Sep 2026). why bother putting GPT-4o here? Also, comparisons with recent chart-specific methods (ChartVerse, Chart-RL, Chart-FR1, chart-to-table/code reasoning pipelines), and stronger benchmarks (ChartQAPro, CharXiv) are needed.
4. Unclear training-data composition: The authors mention additional single-image training examples without specifying their source, quantity, mixing ratio, or reward formulation. Since training primarily involves paired charts while standard chart QA uses single images, the effect of this mixture on general QA performance should be investigated.
5. Insufficient evaluation transparency: The evaluation prompts, scoring protocols, and test subsets are not clearly specified. Chart-Perturb's chart-level independence and potential overlap in PlotQA's underlying numerical data also require clarification.
Questions for authors
Were Table 2's proprietary-model scores adopted directly from ChartSketcher? What evaluation prompts and scoring protocol were used?
3 comments
W2 (2/2-Part)
We therefore introduce a direct visual evidence reading evaluation using PlotQA's structured annotations to assess category identification, series understanding, and numerical extraction. We also include local masking controls targeting question-relevant versus irrelevant visual elements.
Specifically, we construct 900 single-chart questions from 300 PlotQA test charts and compare Qwen3-VL-4B with Chart-SR-4B under identical inference settings.
For target-value reading, we use 300 single-point (1-value) and 300 two-point (2-value) questions, covering 900 target values. In the results, reading accuracy improves from 96.78% to 98.56%.
For question-relevant evidence identification, another 300 questions require identifying the category containing the maximum or minimum value of a specified series. The accuracy increases from 70.67% to 76.00%.
Table: Direct visual evidence evaluation.
| Method | Target-value Reading | Relevant Evidence Identification |
|---|---|---|
| Qwen3-VL-4B | 96.78% (871/900) | 70.67% (212/300) |
| Chart-SR-4B | 98.56% (887/900) | 76.00% (228/300) |
These results illustrate a modest improvement in already strong target-value reading performance, with larger gains in identifying task-relevant extrema.
Accordingly, we refine our main claim to "improved question-conditioned evidence sensitivity", rather than presenting the masking experiment alone as proof of fully faithful visual grounding.
W3 & Q1: Baselines and Evaluation Protocols
We thank the reviewer for highlighting the issues regarding baseline selection and evaluation transparency.
(1) Baseline selection and table organization. Table 2 evaluates general chart QA capabilities, whereas Table 3 focuses on intervention sensitivity and robustness, thus using different benchmarks. We acknowledge that omitting the original Qwen3-VL backbones from Table 2 made the comparison incomplete. We have now added Qwen3-VL-4B/8B results and updated the revised PDF accordingly.
Table: Comparison with original backbones (all values in %, one decimal place).
| Model | ChartQA-H | ChartQA-A | EvoChart-QA | ChartBench | PlotQA | Overall |
|---|---|---|---|---|---|---|
| Qwen3-VL-4B (Base) | 74.2 | 94.2 | 58.3 | 59.3 | 68.6 | 70.9 |
| Qwen3-VL-8B (Base) | 76.7 | 94.9 | 61.4 | 64.9 | 76.9 | 75.0 |
We have included the complete experimental results in the revised PDF.
(2) Closed-source baselines and evaluation protocols. The GPT-4o, Gemini-2.0, and Claude-3.5 results in Table 2 are directly adopted from ChartSketcher (Huang et al., 2025), rather than independently evaluated. They serve as historical references, not as current state-of-the-art baselines. We did not independently reproduce their inference prompts and will explicitly distinguish cited results from our own evaluations in the revision.
We additionally evaluated two recent API-based proprietary-model:
| Model (API) | EvoChart-QA | ChartBench |
|---|---|---|
| Qwen3.8-Flash | 66.7 | 73.9 |
| Qwen4-Max | 74.6 | 77.7 |
(3) Recent baselines and challenging benchmarks. Following the reviewer's suggestion, we additionally included ChartVerse-4B as a recent baseline and extended our evaluation to ChartQAPro. These comparisons aim to provide stronger evidence of Chart-SR's performance and generalization, particularly against models of comparable scale.
Table: Comparison with ChartVerse-4B (all values in %).
| Model | ChartBench | PlotQA |
|---|---|---|
| ChartVerse-4B | 52.9 | 75.0 |
| Chart-SR-4B (Ours) | 68.6 | 78.2 |
W4. Training Data Composition
(1) Single-chart training data.
We clarify that our single-chart training pool contains 1,600 unique chart QA samples, drawn from ChartQA (60%), single-view QA derived from our training CS-Dataset (20%), ChartBench (15%), and FigureQA (5%).
All ChartQA, ChartBench, and FigureQA samples are strictly drawn from their training splits, ensuring separation from evaluation data. Single-chart QA samples help preserve general chart understanding and increase chart-type diversity, as PlotQA / our CS-Dataset cover only four chart types.
(2) Single-chart GRPO ablation.
We further compare the original backbone, single-chart GRPO, and our full model:
Table: Training strategy comparison (%).
| Method | ChartBench | AccAll | AccAC |
|---|---|---|---|
| Qwen3-VL-4B (Base) | 59.33 | 56.0 | 47.0 |
| Single-chart GRPO | 64.29 | 58.0 | 49.0 |
| Chart-SR-4B (Full) | 68.60 | 71.0 | 62.0 |
Single-chart GRPO improves general chart QA performance, while our full model achieves substantially larger gains in both overall accuracy and intervention-sensitive evaluation.
W5. Evaluation Transparency
Thank you for highlighting the need for clearer evaluation details. For standard benchmarks, we follow the official evaluation protocols, including prompts, scoring rules, and test splits. We will explicitly document these settings in the revised manuscript.
For example, the evaluation prompt used for ChartQA is:
Answer the question about the chart image. Return only the final answer, without explanation.
Regarding chart-level independence, the PlotQA benchmark uses the official PlotQA test split, whereas Chart-Perturb is constructed from the remaining samples in the PlotQA training split after extracting our training CS-Dataset (in original paper line 315, we have stated). We further verify that chart and question IDs are disjoint, ensuring no sample-level overlap between training and evaluation sets.
We will clarify this separation procedure and further examine potential overlap in underlying numerical tables in the revision.
W1. Limited Methodological Novelty
We agree that counterfactual learning, visual grounding, and structured reasoning have been studied individually, and have accordingly refined our novelty claims. The key distinctions are summarized below.
Table 1. Comparison with related methods.
Importantly, DeFacto already considers relevant/irrelevant masking and answer preservation. We therefore do not claim that jointly learning sensitivity and invariance is unique to Chart-SR.
Our specific contribution is to exploit the editability of underlying chart data to construct interventions where both charts remain answerable. This enables programmatic verification of both answers, modified elements, and intervention effects, explicitly supervising what the answer should change to and when it should remain unchanged. Unlike CFPO, which examines internal representation sensitivity, Chart-SR evaluates answer responses to semantically controlled input changes.
W2(1/2-Part). Insufficient Evidence for Improved Visual Perception
(1) Visual grounding and perception.
We thank the reviewer for this important observation. Chart-SR's structured reasoning is not merely an output format but decomposes visual evidence perception and utilization into verifiable subtasks:
These tasks involve visual perception, understanding, and reasoning rather than simply following a fixed template. Notably, the format reward accounts for only 5% of the total reward, while the remaining 95% supervises task correctness.
Although we do not explicitly verify every extracted value or evidence region, these verifiable objectives indirectly reinforce the model's ability to perceive and utilize question-relevant visual changes. We clarify that our primary goal is to improve question-conditioned fine-grained visual evidence sensitivity and reasoning, rather than standalone visual recognition.
(2) Zero-shot and few-shot comparisons.
Following the reviewer's suggestion, we evaluated Qwen3-VL-4B-Instruct under zero-shot and four-shot settings against Chart-SR-4B on the same 1,000 Chart-Perturb questions, using identical five-part output instructions and decoding settings.
Both zero-shot and Chart-SR directly receive the original chart, modified chart, and question. The four-shot setting additionally provides four fixed training examples, each containing paired charts, a question, and a complete structured solution. Two examples involve answer changes and two preserve the answer, with no chart overlap with the test set.
We separately evaluate format compliance and two task-level metrics:
Table 2. Zero-shot and few-shot comparisons (%, higher is better).
Four-shot prompting improves format compliance from 10.4% to 99.3%, but JointAcc increases only marginally, while BalancedEffectAcc decreases. Chart-SR achieves substantially higher task-level accuracy.
These results indicate that four demonstrations are sufficient to establish the output format, but format imitation alone cannot reproduce the improvements achieved through Chart-SR training.