- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether charts can serve as intermediate epistemic actions in agentic data analysis rather than merely as final visualization artifacts. The authors first introduce ProbeBench, a controlled benchmark of 200 analytical tasks, revealing a gap between agents' ability to understand charts and their tendency to use them during reasoning. The study also highlights the complementary roles of textual and visual representations across different analytical tasks.
Based on these findings, the authors propose ChartMind, an experience-driven framework that distills analytical and representation-selection knowledge from diverse, conditioned agent trajectories. The resulting experience package is iteratively refined through a validation gate and reused as external memory without updating model parameters.
Experiments across three held-out benchmarks and four LLM backbones demonstrate consistent improvements in analytical accuracy and visualization quality. Ablation studies further examine the contributions of conditioned rollouts, validation gating, and the two types of experience.
Critical strengths
1. Interesting problem formulation and insightful empirical observations. The paper presents an interesting perspective on treating charts as intermediate reasoning actions rather than final presentation artifacts. The controlled study on ProbeBench reveals a meaningful capability–use gap and provides useful evidence of the complementary strengths of textual and visual representations.
2. Consistent improvements across diverse settings. ChartMind demonstrates improvements across three downstream benchmarks and four LLM backbones without parameter updates. The ablation studies also provide evidence for the effectiveness of conditioned rollouts, validation gating, and the complementary roles of analytical and representation experience.
Critical weaknesses
1. Limited methodological novelty and insufficient comparison with simpler alternatives. The proposed framework primarily combines established techniques, including conditioned trajectory generation, LLM-based experience distillation, external memory, and validation-based refinement. While their integration is reasonable, the methodological contribution appears relatively incremental. More importantly, the paper does not establish whether such a complex framework is necessary compared with simpler alternatives, such as rule-based representation selection, direct prompting with visualization guidelines, or lightweight LLM-based routing.
2. Insufficient evidence that charts genuinely function as epistemic actions. The central claim is that charts actively contribute to intermediate reasoning rather than merely improving the final presentation. However, the current evaluation mainly measures final task performance, visualization quality, and chart-generation frequency. These metrics do not directly establish whether agents actually use visual observations to revise their subsequent analytical decisions. Furthermore, the ablation results suggest that general analytical experience accounts for a substantial portion of the performance improvement, making it difficult to isolate the contribution of chart-based reasoning. This weakens the empirical support for the paper's central claim.
Questions for authors
Q1. How can the authors demonstrate that agents actually use intermediate chart observations to guide subsequent reasoning? For example, would removing or masking chart observations while preserving chart-generation actions significantly affect analytical performance?
Q2. How does ChartMind compare with simpler representation-selection strategies, such as predefined rules based on task types, direct prompting with visualization guidelines, or LLM-based selection between textual and visual representations? Such comparisons would help establish the necessity of experience-driven distillation.
Q3. The ablation results show that analysis experience alone accounts for a substantial portion of the overall improvement. To what extent can the gains of ChartMind be attributed specifically to learning when and how to use charts, rather than to general improvements in analytical reasoning?
Q4. How reliable are the step-level attribution labels generated by the trajectory judge? Is there any human validation or agreement analysis supporting its ability to distinguish helpful representation choices from helpful analytical operations?
Q5. ChartMind increases the number of tool-call rounds in the reported trajectory analysis. How do the improvements compare with the additional inference costs, including token consumption, tool executions, and latency? Do the gains remain meaningful under matched computational budgets?
Q6. How sensitive is the learned experience package to the backbone used for distillation? Since the final package is constructed using a single backbone, would distilling experience from a different model produce similar transfer performance?
4 comments
Q3: To what extent are ChartMind’s gains chart-specific rather than general analytical reasoning improvements?
A3: Thank you for raising this point. Thank you for raising this point. We agree that, on top of analysis experience, the marginal Acc. gain of representation experience is small (+0.87; Overall 58.74 to 59.78). However, this marginal value understates its contribution. First, representation experience improves accuracy on its own: the representation-only setting raises Acc. from 46.61 to 49.31 (+2.70), comparable to analysis-only (+3.70). Second, the two types of experience overlap substantially: the sum of their individual gains (6.40) exceeds the combined gain (4.57), so the marginal contribution depends on the order of addition. If representation experience is added first, the marginal Acc. gain of analysis experience is also only +1.87. This overlap is expected, as both are distilled from the same conditioned text-preferred and chart-preferred trajectories. Third, the distinct contribution of representation experience lies mainly in visual evidence: on top of analysis experience, it raises Vis. from 10.43 to 27.61, whereas analysis experience alone improves Vis. by only 2.77. We will clarify this in the final version.
Q4: Concerns about the reliability and discriminative ability of step-level attribution
A4: Step-level attribution involves reflecting on the usefulness of each step in a rollout trajectory. Each step is a single code execution that may include both analysis and representation operations. Following your suggestion, we manually annotate 20 sampled rollout trajectories and assess human–LLM agreement. The results are shown below:
| Human label / LLM judge | Helpful | Ineffective | Harmful | Class agreement |
|---|---|---|---|---|
| Helpful | 47 | 0 | 3 | 94.0% |
| Ineffective | 0 | 2 | 0 | 100.0% |
| Harmful | 2 | 1 | 10 | 76.9% |
| Overall accuracy | — | — | — | 90.8% |
The LLM judge and human annotations agree on 59 out of 65 steps, yielding an overall accuracy of 90.8%. Conditioned on the human label, the class-wise agreement rates for Helpful, Ineffective, and Harmful are 94.0%, 100.0%, and 76.9%, respectively. These results strongly support the reliability of the step-level attribution labels.
Q5: Comparison under matched computational budgets
A5: We align ChartMind's inference costs with those of each corresponding baseline trajectory. The results are shown below:
We provide additional experimental results with ChartMind’s inference budget matched to each corresponding baseline trajectory. The results are shown below:
| Setting | Acc. | Vis. | Overall | Mean input tokens | Mean output tokens | Mean tool calls |
|---|---|---|---|---|---|---|
| Baseline | 46.61 | 7.66 | 54.32 | 73,339 | 9,312 | 8.44 |
| ChartMind | 51.18 | 27.61 | 59.78 | 122,451 | 9,733 | 12.85 |
| ChartMind, budget matched | 50.40 | 20.20 | 58.80 | 75,242 | 8,207 | 8.74 |
Under approximately matched budgets, ChartMind achieves an Overall score of 58.80, exceeding the baseline by 4.48 points and retaining 82% of its full-budget gain. It uses 2.6% more input tokens, 11.9% fewer output tokens, and 3.6% more tool calls than the baseline. Thus, the gains remain meaningful at similar inference costs.
Q6: How sensitive is the learned experience to the LLM backbone used for distillation?
A6: We conduct an ablation study using different LLM backbones for distillation. The results are shown below:
| Builder backbone | Acc. | Vis. | Overall |
|---|---|---|---|
| Baseline | 46.61 | 7.66 | 54.32 |
| Claude Sonnet 5 | 49.93 | 21.00 | 59.41 |
| DeepSeek V4.1 Flash | 51.85 | 25.71 | 60.89 |
| GPT-5.6 Luna | 48.86 | 25.05 | 59.28 |
| GPT-5.6 Sol | 51.18 | 27.61 | 59.78 |
ChartMind significantly outperforms the baseline across all four builder models. DeepSeek V4.1 Flash achieves the highest Overall score of 60.89, while GPT-5.6 Luna yields the lowest score of 59.28. The highest and lowest scores differ by 1.61 points. These results show that ChartMind effectively improves performance across different backbone models.
Thanks again for your important questions. Hope the additional evidence could addresses your concerns, and we would appreciate it if you could reassess our work and increase our rating.
I thank the authors for their response. My concerns have been largely addressed. Although I do not consider the methodological novelty of this paper to be very high, the insights and findings are valuable. I have adjusted my score to 3.
Thank you for your thoughtful feedback and for reassessing our work. We are glad that our response has largely addressed your concerns, and we appreciate your recognition of the value of our insights and findings.
Thank you again for your time and effort throughout the review process.
Response to Reviewer: wxwa
Thanks for your careful and valuable comments. We will refine the paper following your suggestions. We provide detailed responses to your concerns below.
Q1: Evidence that the agent uses intermediate chart observations to guide subsequent reasoning
A1: Thank you for this important question. Following your suggestion, we conduct chart-masking experiments on DAComp-DA with GPT-5.6-Luna:
Compared with default, chart masked keeps the same trajectory but removes chart access when generating the final answer. Acc. drops from 51.18 to 46.77, close to the baseline (46.61), indicating that chart observations carry evidence required for correct conclusions that is not fully captured in the textual trajectory. Compared with chart masked, blind further removes chart access during analysis. This lowers Vis. (26.43 to 16.33) and Overall (59.25 to 57.30), while Acc. remains similar (46.81). Thus, viewing charts during analysis mainly shapes how the agent selects and constructs subsequent visual evidence. In summary, charts provide critical evidence for analytical reasoning and decision-making.
We further conduct a trajectory attribution analysis on the baseline and ChartMind default. We measure the numbers of text and chart observations, as well as the numbers of observations that support the final answer.
ChartMind adds 48 chart observations and 26 helpful chart observations. Helpful chart evidence increases from 11 to 37, a 3.36× increase, while both text measures remain similar. Together with the masking results, this finding shows that experience learning enables agents to use visual evidence to improve downstream analysis.
Q2: Comparison with simple representation-selection strategies
A2: Following your suggestion, we compare learned representation experience with three simpler alternatives. For predefined rules, we manually define a set of rules and directly prompt the Agent. For the LLM-based method, we use gpt-5.6-sol as an expert to generate task-specific experience for each trajectory, and prompt the main agent. The experimental results are as follows:
ChartMind outperforms all comparison methods, achieving an Overall score of 59.78. Predefined rules and the LLM-based method improve the baseline Overall score to 57.10 and 57.43, respectively. These results demonstrate the effectiveness of our method.