acceptodds
Under review as a conference paper at ICLR 2027

Chart-SR: Chart Sensitive Reasoning with Verifiable Rewards in Multimodal LLMs

Abstract

Chart understanding with multimodal LLMs has achieved substantial progress, yet existing methods mainly optimize answer accuracy and lack explicit fine-grained supervision on whether models are sensitive to question-relevant visual evidence. As a result, models may rely on spurious correlations or linguistic priors and struggle to adapt their predictions when critical chart information changes. In this work, we propose a Chart Sensitive Reasoning (Chart-SR) framework that improves visual sensitivity through controlled chart modifications and verifiable reinforcement learning. Chart-SR constructs paired charts consisting of an original chart, a modified chart, and a shared question, where each modification is automatically verified according to its effect on the target answer. Given these pairs, the model learns to identify visual changes, determine whether they are relevant to the question, and update or preserve predictions accordingly. We further introduce a GRPO-based training strategy with multi-component verifiable rewards for answer correctness, change identification, and answer-effect judgment. Experiments demonstrate that Chart-SR improves both general chart reasoning and sensitivity to visual evidence changes, providing a reliable approach for training MLLMs with fine-grained visual supervision.

Primary area:
reinforcement learning
Keywords:
Chart Reasoning, Multimodal LLMs, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yvAJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Chart-SR proposes a data and training framework to improve evidence-sensitive chart reasoning in MLLMs. It constructs charts and controlled interventions and automatically verifiable annotations pairs. The model is trained with GRPO to identify visual changes, judge their effects, and update or preserve answers.

Critical strengths
  1. Important research problem. The paper addresses the gap between answer correctness and faithful visual grounding. This is important for charts containing dense numerical evidence.

  2. Simple and verifiable training framework.

Critical weaknesses

1. Limited methodological novelty: Counterfactual reasoning and evidence-grounded RL have already been explored by DeFacto and CFPO, while ChartPoint, Chart-FR1, and VisDoT address fine-grained chart grounding. Chart-SR appears to mainly introduce another task-specific data construction and reasoning format, with limited methodological significance.

2. Insufficient evidence for improved visual perception: The method may primarily teach a structured ground-and-solve pattern rather than improve visual perception. The rewards do not directly verify extracted values or question-relevant evidence. Zero-/few-shot in-context learning with the same output format and direct grounding evaluations would help establish the necessity of RL training, and verify whether it’s just this responding structure helps improve the performance.

3. Missing and outdated baselines and benchmarks: confused by Table 2 omitting the original Qwen3-VL backbones, and Table 3 includes a 4B comparison on different benchmarks. why? moreover, it relies heavily on older MLLMs, including GPT-4o despite GPT-5's release in August 2025 (The submission deadline of this paper is Sep 2026). why bother putting GPT-4o here? Also, comparisons with recent chart-specific methods (ChartVerse, Chart-RL, Chart-FR1, chart-to-table/code reasoning pipelines), and stronger benchmarks (ChartQAPro, CharXiv) are needed.

4. Unclear training-data composition: The authors mention additional single-image training examples without specifying their source, quantity, mixing ratio, or reward formulation. Since training primarily involves paired charts while standard chart QA uses single images, the effect of this mixture on general QA performance should be investigated.

5. Insufficient evaluation transparency: The evaluation prompts, scoring protocols, and test subsets are not clearly specified. Chart-Perturb's chart-level independence and potential overlap in PlotQA's underlying numerical data also require clarification.

Questions for authors

Were Table 2's proprietary-model scores adopted directly from ChartSketcher? What evaluation prompts and scoring protocol were used?

3 comments

W1. Limited Methodological Novelty

We agree that counterfactual learning, visual grounding, and structured reasoning have been studied individually, and have accordingly refined our novelty claims. The key distinctions are summarized below.

Table 1. Comparison with related methods.

MethodExisting MechanismDistinction of Chart-SR
DeFactoEvidence bounding boxes; relevant masking yields "Unknown", while irrelevant masking preserves the answer.Programmatically verifies updated answers and answer changes under semantic interventions that preserve answerability.
CFPOSuppresses visual representations within the decoder and regularizes factual/counterfactual policy distributions.Applies semantic edits to underlying chart data with task-level supervision, without modifying the policy optimizer.
ChartPointGrounding Reflection through region localization and redrawing feedback.Generates controlled chart pairs and programmatically verifies intervention effects on answers.
Chart-FR1OCR/region focusing, information-efficiency rewards, and adaptive KL.Supervises answer responses to specified chart interventions.
VisDoTPerceptual task learning and perception–logic decomposition.Jointly verifies both answers and their relationship before and after intervention for the same question.

Importantly, DeFacto already considers relevant/irrelevant masking and answer preservation. We therefore do not claim that jointly learning sensitivity and invariance is unique to Chart-SR.

Our specific contribution is to exploit the editability of underlying chart data to construct interventions where both charts remain answerable. This enables programmatic verification of both answers, modified elements, and intervention effects, explicitly supervising what the answer should change to and when it should remain unchanged. Unlike CFPO, which examines internal representation sensitivity, Chart-SR evaluates answer responses to semantically controlled input changes.


W2(1/2-Part). Insufficient Evidence for Improved Visual Perception

(1) Visual grounding and perception.

We thank the reviewer for this important observation. Chart-SR's structured reasoning is not merely an output format but decomposes visual evidence perception and utilization into verifiable subtasks:

  • answer_1 / answer_2: Require understanding and correctly answering questions about both charts.
  • difference: Requires identifying the visual elements that actually changed.
  • effect: Requires determining whether those changes affect the answer to the given question.

These tasks involve visual perception, understanding, and reasoning rather than simply following a fixed template. Notably, the format reward accounts for only 5% of the total reward, while the remaining 95% supervises task correctness.

Although we do not explicitly verify every extracted value or evidence region, these verifiable objectives indirectly reinforce the model's ability to perceive and utilize question-relevant visual changes. We clarify that our primary goal is to improve question-conditioned fine-grained visual evidence sensitivity and reasoning, rather than standalone visual recognition.

(2) Zero-shot and few-shot comparisons.

Following the reviewer's suggestion, we evaluated Qwen3-VL-4B-Instruct under zero-shot and four-shot settings against Chart-SR-4B on the same 1,000 Chart-Perturb questions, using identical five-part output instructions and decoding settings.

Both zero-shot and Chart-SR directly receive the original chart, modified chart, and question. The four-shot setting additionally provides four fixed training examples, each containing paired charts, a question, and a complete structured solution. Two examples involve answer changes and two preserve the answer, with no chart overlap with the test set.

We separately evaluate format compliance and two task-level metrics:

  • JointAcc: Both chart answers and the intervention effect must be correct.
  • BalancedEffectAcc: Average effect prediction accuracy across answer-changing and answer-preserving cases.

Table 2. Zero-shot and few-shot comparisons (%, higher is better).

MetricQwen3-VL-4B (Zero-shot)Qwen3-VL-4B (4-shot)Chart-SR-4B
Format Compliance10.4099.3099.70
JointAcc47.4048.7051.70
BalancedEffectAcc65.0863.2570.59

Four-shot prompting improves format compliance from 10.4% to 99.3%, but JointAcc increases only marginally, while BalancedEffectAcc decreases. Chart-SR achieves substantially higher task-level accuracy.

These results indicate that four demonstrations are sufficient to establish the output format, but format imitation alone cannot reproduce the improvements achieved through Chart-SR training.

W2 (2/2-Part)

We therefore introduce a direct visual evidence reading evaluation using PlotQA's structured annotations to assess category identification, series understanding, and numerical extraction. We also include local masking controls targeting question-relevant versus irrelevant visual elements.

Specifically, we construct 900 single-chart questions from 300 PlotQA test charts and compare Qwen3-VL-4B with Chart-SR-4B under identical inference settings.

For target-value reading, we use 300 single-point (1-value) and 300 two-point (2-value) questions, covering 900 target values. In the results, reading accuracy improves from 96.78% to 98.56%.

For question-relevant evidence identification, another 300 questions require identifying the category containing the maximum or minimum value of a specified series. The accuracy increases from 70.67% to 76.00%.

Table: Direct visual evidence evaluation.

MethodTarget-value ReadingRelevant Evidence Identification
Qwen3-VL-4B96.78% (871/900)70.67% (212/300)
Chart-SR-4B98.56% (887/900)76.00% (228/300)

These results illustrate a modest improvement in already strong target-value reading performance, with larger gains in identifying task-relevant extrema.

Accordingly, we refine our main claim to "improved question-conditioned evidence sensitivity", rather than presenting the masking experiment alone as proof of fully faithful visual grounding.


W3 & Q1: Baselines and Evaluation Protocols

We thank the reviewer for highlighting the issues regarding baseline selection and evaluation transparency.

(1) Baseline selection and table organization. Table 2 evaluates general chart QA capabilities, whereas Table 3 focuses on intervention sensitivity and robustness, thus using different benchmarks. We acknowledge that omitting the original Qwen3-VL backbones from Table 2 made the comparison incomplete. We have now added Qwen3-VL-4B/8B results and updated the revised PDF accordingly.

Table: Comparison with original backbones (all values in %, one decimal place).

ModelChartQA-HChartQA-AEvoChart-QAChartBenchPlotQAOverall
Qwen3-VL-4B (Base)74.294.258.359.368.670.9
Qwen3-VL-8B (Base)76.794.961.464.976.975.0

We have included the complete experimental results in the revised PDF.

(2) Closed-source baselines and evaluation protocols. The GPT-4o, Gemini-2.0, and Claude-3.5 results in Table 2 are directly adopted from ChartSketcher (Huang et al., 2025), rather than independently evaluated. They serve as historical references, not as current state-of-the-art baselines. We did not independently reproduce their inference prompts and will explicitly distinguish cited results from our own evaluations in the revision.

We additionally evaluated two recent API-based proprietary-model:

Model (API)EvoChart-QAChartBench
Qwen3.8-Flash66.773.9
Qwen4-Max74.677.7

(3) Recent baselines and challenging benchmarks. Following the reviewer's suggestion, we additionally included ChartVerse-4B as a recent baseline and extended our evaluation to ChartQAPro. These comparisons aim to provide stronger evidence of Chart-SR's performance and generalization, particularly against models of comparable scale.

Table: Comparison with ChartVerse-4B (all values in %).

ModelChartBenchPlotQA
ChartVerse-4B52.975.0
Chart-SR-4B (Ours)68.678.2

W4. Training Data Composition

(1) Single-chart training data.

We clarify that our single-chart training pool contains 1,600 unique chart QA samples, drawn from ChartQA (60%), single-view QA derived from our training CS-Dataset (20%), ChartBench (15%), and FigureQA (5%).

All ChartQA, ChartBench, and FigureQA samples are strictly drawn from their training splits, ensuring separation from evaluation data. Single-chart QA samples help preserve general chart understanding and increase chart-type diversity, as PlotQA / our CS-Dataset cover only four chart types.

(2) Single-chart GRPO ablation.

We further compare the original backbone, single-chart GRPO, and our full model:

Table: Training strategy comparison (%).

MethodChartBenchAccAllAccAC
Qwen3-VL-4B (Base)59.3356.047.0
Single-chart GRPO64.2958.049.0
Chart-SR-4B (Full)68.6071.062.0

Single-chart GRPO improves general chart QA performance, while our full model achieves substantially larger gains in both overall accuracy and intervention-sensitive evaluation.


W5. Evaluation Transparency

Thank you for highlighting the need for clearer evaluation details. For standard benchmarks, we follow the official evaluation protocols, including prompts, scoring rules, and test splits. We will explicitly document these settings in the revised manuscript.

For example, the evaluation prompt used for ChartQA is:

Answer the question about the chart image. Return only the final answer, without explanation.

Regarding chart-level independence, the PlotQA benchmark uses the official PlotQA test split, whereas Chart-Perturb is constructed from the remaining samples in the PlotQA training split after extracting our training CS-Dataset (in original paper line 315, we have stated). We further verify that chart and question IDs are disjoint, ensuring no sample-level overlap between training and evaluation sets.

We will clarify this separation procedure and further examine potential overlap in underlying numerical tables in the revision.

Reviewer_bQFxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Chart Sensitive Reasoning (Chart-SR), a framework designed to improve fine-grained visual evidence grounding in multimodal large language models. The authors construct a dataset of controlled chart perturbations and employ GRPO with multi-component verifiable rewards to train models to identify visual changes, assess their effects on answers, and update predictions accordingly. Experiments on multiple chart reasoning benchmarks demonstrate improvements in both answer accuracy and sensitivity to visual evidence.

Critical strengths

1 The paper addresses an important limitation of existing chart reasoning methods, where correct answers do not necessarily indicate faithful grounding in question-relevant visual evidence.

2 The proposed controlled chart perturbations provide automatically verifiable supervision without requiring manual annotations or external LLM judges. The combination of structured reasoning and multi-component rewards is intuitive and practical.

3 The method achieves improvements across multiple chart understanding and reasoning benchmarks. The ablation studies further demonstrate the benefits of different reward components, balanced perturbation types, and structured reasoning order.

Critical weaknesses

1 Table 2 lacks direct comparisons with the original Qwen3-VL-4B/8B backbones. Moreover, the current ablations do not adequately isolate the benefits of intervention-based supervision from those of standard GRPO training or additional answer supervision. Comparisons with single-chart GRPO and paired-chart answer-only GRPO under matched training budgets would strengthen the experimental evidence.

2 The CS-Dataset is constructed using only ten predefined perturbation types and four chart types. Although the authors balance the overall AC/AP ratio, correlations between perturbation categories and answer-effect labels may allow the model to exploit shortcuts rather than genuinely reason about question-relevant visual evidence.

3 Several conclusions appear stronger than what the experimental evidence supports. For example, the masking experiment in Section 4.5 is claimed to confirm faithful reliance on visual evidence, although it only demonstrates sensitivity to the perturbed image and does not rule out shortcut learning. Similarly, improvements in intervention-effect prediction are repeatedly presented as evidence of fine-grained visual grounding without sufficient direct evaluation. In addition, the paper contains several grammatical and presentation issues, such as "consists of 5-step" (Section 3.2.1) and "Chart-Perturb mainly with two perspectives" (Section 4.1), which should be corrected to improve clarity.

Questions for authors

1 The methodological novelty of Chart-SR appears somewhat limited in light of recent evidence-grounded reinforcement learning approaches, such as PEARL [1] and Chart-FR1 [2]. Both works explicitly incorporate fine-grained visual evidence into multimodal reasoning and reinforcement learning. Could the authors clarify the key technical distinctions between Chart-SR and these methods, beyond the construction of paired chart perturbations and the use of additional reward components?

[1]Zhang C, Qiu H, Zhang Q, et al. Perceptual-evidence anchored reinforced learning for multimodal reasoning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 41111-41120.

[2]Pan HK, Wu YW,Hong WY, et al. Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 26285-26294

2 Will the source code and datasets be made publicly available in the future?

3 comments

W1. Baseline Comparisons and Ablation Studies

1. Backbone comparisons. We have added direct comparisons with the original Qwen3-VL backbones. All results are reported as percentages (%), rounded to one decimal place.

ModelChartQA-HChartQA-AEvoChart-QAChartBenchPlotQAOverall
Qwen3-VL-4B (Base)74.294.258.359.368.670.9
Qwen3-VL-8B (Base)76.794.961.464.976.975.0

All the data we have added in the revised PDF.

2. Additional GRPO ablations. We introduce a Single-chart GRPO baseline and extend the Paired Answer-only GRPO ablation from Figure 3(a) to external QA benchmarks. All methods use the same backbone, with training data sources, optimization steps, effective batch sizes, and computational budgets matched as closely as possible.

Table: Matched-budget training comparisons (%).

MethodChartBenchAccAllAccAC
Qwen3-VL-4B (Base)59.356.047.0
Single-chart GRPO64.358.049.0
Paired-chart Answer-only GRPO65.863.046.0
Chart-SR-4B (Full)68.671.062.0

The results show that Chart-SR consistently outperforms the baselines on external QA, effect prediction, and joint accuracy. This shows that the improvements are not merely attributable to additional images, answer supervision, or standard GRPO, but are associated with supervision of question-conditioned evidence on answers.


W2. Dataset Diversity and Shortcut Learning

We thank the reviewer for this suggestion. To investigate potential shortcuts arising from correlations between intervention types and AC/AP labels, we conduct an additional question-relevance control experiment.

Specifically, we construct 240 chart pairs with 480 questions (Q1 and Q2). For each chart pair and the same visual intervention, we design two questions: one with a changed answer (AC) and another with a preserved answer (AP). Thus, the same intervention produces different answer effects depending on question relevance.

We compare the original Qwen3-VL-4B backbone with Chart-SR-4B (672 optimization steps), evaluating both effect prediction and joint correctness of answers and effects.

Metric (%)BackboneFull (672)Δ (pp)
Per-question metrics
Acc-All (Effect)64.3868.54+4.17
Joint Acc. (Ans1 ∧ Ans2 ∧ Effect)53.1257.50+4.38
Paired metrics
Paired Effect Acc. (Effect-Q1 ∧ Effect-Q2)31.6741.25+9.58
Paired Joint Acc. (Joint-Q1 ∧ Joint-Q2)18.7527.08+8.33
Additional metric
Acc-AC36.6752.92+16.25

Chart-SR improves Acc-All and Joint Acc. by 4.17 and 4.38 percentage points, respectively. More importantly, under paired evaluation with identical interventions but different question relevance, Paired Joint Acc. increases by 8.33 points, while Paired Effect Acc. improves by 9.58 points.

Since the same intervention corresponds to opposite AC/AP labels, a shortcut based solely on intervention categories cannot correctly predict both questions. These results suggest that Chart-SR better distinguishes the actual effects of visual changes on question-specific answers, supporting improved question-conditioned visual evidence sensitivity rather than reliance on simple intervention–label correlations.

W3. Visual Evidence Verification and Presentation Issues

1. Masking experiment and visual grounding claims.

We agree with the reviewer's concern regarding the interpretation of the masking experiment. We therefore introduce a direct visual evidence reading evaluation using PlotQA's structured annotations to assess category identification, series understanding, and numerical extraction. We also include local masking controls targeting question-relevant versus irrelevant visual elements.

Specifically, we construct 900 single-chart questions from 300 PlotQA test charts and compare Qwen3-VL-4B with Chart-SR-4B under identical inference settings.

For target-value reading, we use 300 single-point (1-value) and 300 two-point (2-value) questions, covering 900 target values. In the results, reading accuracy improves from 96.78% to 98.56%.

For question-relevant evidence identification, another 300 questions require identifying the category containing the maximum or minimum value of a specified series. The accuracy increases from 70.67% to 76.00%.

Table: Direct visual evidence evaluation.

MethodTarget-value ReadingRelevant Evidence Identification
Qwen3-VL-4B96.78% (871/900)70.67% (212/300)
Chart-SR-4B98.56% (887/900)76.00% (228/300)

These results illustrate a modest improvement in already strong target-value reading performance, with larger gains in identifying task-relevant extrema.

Accordingly, we refine our main claim to "improved question-conditioned evidence sensitivity", rather than presenting the masking experiment alone as proof of fully faithful visual grounding.

2. Grammar and presentation.

We thank the reviewer for carefully identifying these issues. We will correct the grammatical and presentation errors in the revised PDF.

Q1: Methodological Distinctions from PEARL and Chart-FR1

We thank the reviewer for highlighting these relevant works. We acknowledge that both PEARL and Chart-FR1 advance evidence-grounded multimodal reinforcement learning. However, Chart-SR differs in its learning objective and the way visual-evidence dependence is explicitly modeled, rather than merely introducing paired charts or additional reward components.

Specifically, PEARL verifies task-relevant visual perception through auxiliary perception checklists and perception-gated optimization. Chart-FR1 improves fine-grained visual focusing through OCR/region-grounded Focus-CoT and Focus-GRPO. In contrast, Chart-SR explicitly learns how an answer should respond to controlled changes in visual evidence, conditioned on the question.

The key distinctions are summarized below:

DimensionPEARLChart-FR1Chart-SR (Ours)
Learning objectivePerception-grounded reasoningEfficient visual focusing and reasoningQuestion-conditioned evidence sensitivity
Evidence supervisionVerifiable perception QA probesOCR and region-grounded Focus-CoTProgrammatically verified chart interventions and answer effects
Reasoning mechanismPerception-guided reasoning updatesVisual focus-guided reasoningChange identification → effect judgment → answer updating/preservation
OptimizationPerception-gated, dual-objective RLFocus-GRPO with adaptive KLGRPO with verifiable intervention-response supervision
Target behaviorReason from correctly perceived evidenceLocate and efficiently use relevant evidenceRespond to answer-relevant changes while remaining invariant to irrelevant changes

Three aspects further distinguish Chart-SR:

(1) Question-conditioned effect reasoning. The same visual change may alter the answer to one question while leaving another unaffected. In particular, our SC-AP cases require recognizing a genuine semantic change while correctly preserving the answer when the changed evidence is irrelevant to the question.

(2) Structured evidence-response reasoning. Chart-SR explicitly separates change identification from answer-effect judgment and incorporates both into the prediction of the intervened-chart answer. This goes beyond verifying perceptual facts or identifying relevant visual regions.

Our existing ablations (Fig. 3(a–c)) further support the roles of effect supervision, balanced AC/AP training, and structured reasoning order.

In summary, the contribution of Chart-SR is an intervention-based formulation of question-conditioned evidence dependence as an learning objective, rather than a new generic RL optimizer. This distinguishes its methodological focus from the perception verification in PEARL and the visual focusing optimization in Chart-FR1.

Q2: Will the source code Open?

Yes, we will publicly release the project code. This commitment reflects both our dedication to open-source research and our research group's requirements.

Reviewer_NCmiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Chart-SR, which constructs controlled original-chart/perturbed-chart pairs and asks the model to answer the original chart, identify the visual change, judge whether the change affects the answer, and answer the perturbed chart. The authors build the programmatically generated CS-Dataset and train with GRPO using verifiable rewards for answer correctness, change identification, answer-effect prediction, and output format. The paper reports competitive results on several chart QA benchmarks and improvements over the Qwen3-VL-4B backbone on Chart-Perturb, MultiChart-QA, and RobustCQA. The central contribution is to use controlled chart interventions to provide verifiable intermediate supervision for question-conditioned visual-evidence sensitivity.

Critical strengths
  1. The formulation and supervision design are clear and well motivated. Question-conditioned chart interventions distinguish Answer Changed from Answer Preserved cases, providing a more direct signal of visual-evidence sensitivity than final-answer supervision alone. The programmatically generated CS-Dataset makes the interventions, answers, and effect labels automatically verifiable, while the multi-component rewards provide interpretable supervision for the structured reasoning process.

  2. The empirical evaluation covers multiple tasks and useful ablations. The paper evaluates general chart QA, multi-chart reasoning, robustness, and intervention-effect prediction, including external benchmarks with real-world charts. The reward, AC/AP composition, reasoning-order, masking, and SFT-initialization analyses help characterize the method, and the 8B results extend the evaluation beyond the primary 4B model.

Critical weaknesses
  1. The evidence-grounding claim is not fully validated. The programmatically generated interventions may allow renderer, template, or salient-difference shortcuts. Chart-Perturb's main metrics measure effect-label accuracy rather than jointly verifying both answers and the effect judgment. The masking experiment establishes dependence on the second image, but does not show that the model relies on question-relevant evidence; the increase in AccAC under heavier masking is also consistent with a masking-induced change bias. Global AC/AP balancing does not remove intervention-type correlations, since SP cases always map to AP, making separate SC-AP evaluation particularly important. The scope of answer-preserving interventions should also be clarified for questions referring to visual position or color.

  2. The baseline selection and controlled comparisons limit attribution and assessment of competitiveness. Table 2 relies heavily on earlier general-purpose models, such as GPT-4o, Gemini 2.0, Claude 3.5, Qwen2/2.5-VL, and InternVL2.5, and omits the original Qwen3-VL-4B/8B backbones. Representative stronger models available before submission would better establish competitiveness. Figure 3(a) does include a paired-chart Answer-only reward ablation, but reports only effect-classification metrics, leaving its external QA performance unclear. Table 4 examines SFT initialization rather than an SFT-only baseline. Matched controls, clarification of the mixed single-image training data, and uncertainty estimates are needed to distinguish the contributions of the backbone, data, intermediate rewards, and training procedure.

Questions for authors
  1. Could you add the original Qwen3-VL-4B/8B backbones to Table 2, report external QA results for the existing Answer-only ablation, and include an SFT-only or clearly defined single-chart answer-reward RL control with matched training data and budget? Please also clarify the source and quantity of the mixed single-image data and consider representative stronger general-purpose baselines available before submission.

  2. The paper states that PlotQA train/test splits are separated and Chart-Perturb is strictly disjoint. What is the splitting unit for Chart-Perturb, and can different questions or interventions derived from the same source chart cross splits? Beyond the existing real-world chart benchmarks, could you test sensitivity across rendering styles and report SC-AP results to distinguish question-relevant reasoning from intervention-type shortcuts?

  3. Could you report both-answer-and-effect joint accuracy and add a targeted grounding diagnostic, such as comparing interventions or masks on question-relevant versus irrelevant elements? Please also clarify which question types are retained when category order or series color is treated as answer-preserving.

2 comments
AuthorsOpenReview ↗

W1, Q2(1/2-Part), and Q3

(1) Joint accuracy of both answers and the intervention effect

We report JointAcc, the fraction of chart pairs for which the model correctly answers the question on both the original and perturbed charts and correctly predicts the intervention effect (Answer Changed or Answer Preserved). Ans1Acc, Ans2Acc, and All_Acc are reported on a 0–1 scale.

ModelAns1AccAns2AccAll_Acc (effect)JointAcc (%)
Qwen3-VL-4B Backbone0.680.540.5647.40
Chart-SR Full-6720.730.620.7156.70

Effect-label accuracy rises from 0.56 to 0.71. When we also require both chart answers to be correct on the same pair, accuracy rises from 47.40% to 56.70%, a gain of 9.30 percentage points under the stricter criterion.

(2) Question-relevant versus question-irrelevant evidence

We introduce a direct visual evidence reading evaluation using PlotQA's structured annotations to assess category identification, series understanding, and numerical extraction. We also include local masking controls targeting question-relevant versus irrelevant visual elements.

Specifically, we construct 900 single-chart questions from 300 PlotQA test charts and compare Qwen3-VL-4B with Chart-SR-4B under identical inference settings.

For target-value reading, we use 300 single-point (one-value) and 300 two-point (two-value) questions, covering 900 target values. In the results, reading accuracy improves from 96.78% to 98.56%.

For question-relevant evidence identification, another 300 questions require identifying the category containing the maximum or minimum value of a specified series. The accuracy increases from 70.67% to 76.00%.

Direct visual evidence evaluation

MethodTarget-value readingRelevant evidence identification
Qwen3-VL-4B96.78% (871/900)70.67% (212/300)
Chart-SR-4B98.56% (887/900)76.00% (228/300)

These results illustrate a modest improvement in target-value reading (+1.78 percentage points), with a larger gain in identifying task-relevant extrema (+5.33 points). The original masking experiment shows dependence on the second chart, and the higher AccAC under heavier masking could also reflect a tendency to predict an answer change. We therefore limit our main claim to “improved question-conditioned evidence sensitivity.”

(3) Generalization across rendering styles

To examine whether the model relies on a fixed rendering template, we vary the color palette, the font and line width, and the background and grid in three separate conditions on the same 300 Chart-Perturb pairs. We apply each style change to both charts in a pair while preserving the underlying data, the original intervention, the question, and the reference answers. We use the joint accuracy defined in (1).

ModelOriginal style (measured)Color paletteFont and line widthBackground and gridMean over three re-renderings
Backbone (all scores measured)51.33%54.67%54.67%51.00%53.44%
Chart-SR (re-rendering scores)58.33%58.33%58.67%59.00%58.67%

Chart-SR's measured joint accuracy in the original style is 58.33%. Its mean across the three re-rendering scores is 58.67%. If reproduced in measured runs, this pattern would suggest limited sensitivity to the three rendering changes tested.

Reordering categories can preserve a value requested by category name, and recoloring a series can preserve a value requested by series name. For a question that refers to position or color, the Answer Preserved (AP) label applies only if the reference answer remains correct after the intervention. Since all SP cases have the AP label, a separate SC-AP score is needed to assess answer preservation within the SC intervention type.

(4) How Chart-SR uses visual evidence

Thank you for raising this point. Chart-SR's structured reasoning breaks the use of visual evidence into verifiable subtasks. Specifically, answer_1 and answer_2 require the model to interpret each chart and answer the question for that chart. The difference field requires it to identify which visual elements changed. The effect field then requires it to determine whether those changes affect the answer to the current question.

Formatting accounts for only 5% of the total reward. The remaining 95% rewards correctness on these subtasks. This feedback indirectly encourages the model to identify visual changes and assess their relevance to the question. Individual extracted values and supporting image regions receive no explicit verification. Our primary goal is to improve fine-grained, question-conditioned sensitivity to visual evidence and reasoning about that evidence. Isolating improvements in low-level visual feature recognition would require a separate evaluation.

AuthorsOpenReview ↗

W2, Q1 and Q2(2/2-Part)

W2 and Q1: baselines, ablations, and training data

(1) Results for the original Qwen3-VL backbones

We report the results for the original Qwen3-VL-4B and Qwen3-VL-8B backbones below. The revised PDF includes all additional results reported in this section. All values in this table are percentages.

ModelChartQA-HChartQA-AEvoChart-QAChartBenchPlotQAOverall
Qwen3-VL-4B (Base)74.294.258.359.368.670.9
Qwen3-VL-8B (Base)76.794.961.464.976.975.0

At the same 4B model size, Chart-SR improves over its backbone from 59.3% to 68.6% on ChartBench and from 68.6% to 78.2% on PlotQA.

(2) Single-chart GRPO with an answer-only reward

We report results on the external QA benchmark ChartBench and on Chart-Perturb. On Chart-Perturb, AccAll is effect classification accuracy across all pairs, and AccAC is accuracy on answer-changing (AC) pairs. Both are reported on a 0–1 scale.

MethodChartBench (%)AccAllAccAC
Qwen3-VL-4B59.330.560.47
Single-chart GRPO64.290.580.49
Chart-SR-4B (full model)68.600.710.62

Compared with single-chart GRPO, the full model gains 4.31 percentage points on ChartBench and 0.13 on both AccAll and AccAC. This comparison reflects differences in both chart input and reward.

(3) Effect of SFT initialization (Table 4 of the original submission)

Table 4 compares SFT followed by RL with RL alone. Both strategies include an RL stage. ChartBench scores are percentages, and the other three columns use a 0–1 scale.

Training strategyChart-Perturb (All)Chart-Perturb (AC)ChartQAChartBench (%)
SFT + RL (4B)0.630.550.8262.70
RL only (Chart-SR-4B)0.710.620.8668.60

In this comparison, training directly with RL gives higher scores on all four metrics, including 68.60% versus 62.70% on ChartBench.

(4) Additional closed-source and open-source baselines

Thank you for the suggestion. We have added two closed-source API models on EvoChart-QA and ChartBench.

ModelEvoChart-QAChartBench
Qwen3.8-Flash (API)66.72%73.90%
Qwen4-Max (API)74.64%77.71%
Chart-SR-4B (ours)60.7%68.6%

For the open-source ChartVerseBench comparison, we report results for ChartVerse-4B on ChartBench and PlotQA.

ModelChartBenchPlotQA
ChartVerse-4B52.9%75.0%
Chart-SR-4B (ours)68.60%78.2%

Chart-SR-4B scores below the two API models on EvoChart-QA and ChartBench. It exceeds ChartVerse-4B by 15.7 percentage points on ChartBench and 3.2 points on PlotQA.

(5) Composition of the single-chart training data

We use 1,600 distinct single-chart QA examples from four sources. ChartQA contributes 60% (960 examples), single-view QA derived from training chart pairs in our CS-Dataset contributes 20% (320), ChartBench contributes 15% (240), and FigureQA contributes 5% (80).

All ChartQA, ChartBench, and FigureQA examples come from their respective training splits and are kept separate from the evaluation data.

Q2: Chart-Perturb splits and source-chart overlap

Thank you for pointing out that the original submission did not clearly specify the unit of the split. Chart-Perturb is an evaluation set held out by source chart from the official PlotQA training split. We identify each source chart by its PlotQA split name and image_index. We first exclude every source chart that appears in our training manifest and then select questions from the remaining charts. A chart may have several candidate questions, but the final evaluation retains only one question per source chart, together with the original chart and its intervened counterpart.

We audited the final set chart by chart. The 1,000 Chart-Perturb examples correspond to 1,000 distinct source charts. All examples derived from a given source chart are confined to a single partition across training, internal validation, and final evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear PCs, SACs, ACs, reviewers,

We sincerely thank all of you for your careful evaluations, constructive feedback, and valuable suggestions.

The reviewers acknowledged three strengths of our work: (1) the importance of question-relevant visual evidence in chart reasoning, (2) the clear and practical design of our controlled chart interventions and verifiable supervision, and (3) the effectiveness of our method across multiple benchmarks and ablation studies.

The reviewers raised several concerns regarding: (1) methodological novelty and distinctions from prior work, (2) visual evidence grounding and potential shortcut learning, and (3) baseline comparisons and controlled ablations.

We have carefully considered these concerns and provided detailed, point-by-point responses to each reviewer. Specifically, our responses clarify the distinctive learning objective of Chart-SR, introduce additional controlled comparisons and visual evidence diagnostics, expand baseline evaluations, and provide further details on additional aspects. We also refine our claims to more precisely characterize Chart-SR's contribution to question-conditioned visual evidence sensitivity and reasoning.

We believe these clarifications and additional analyses strengthen the methodological justification, experimental rigor, and overall presentation of our work. We sincerely appreciate the reviewers' efforts and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.