acceptodds
Under review as a conference paper at ICLR 2027

Charts as Epistemic Actions for Experience-Driven Agentic Data Analysis

Abstract

Data agents built on ReAct-style loops now execute complex analytical workflows over structured data, yet their intermediate reasoning still runs almost entirely through text. Human analysts instead move between numbers, tables, and charts, reading visual evidence to expose trends and anomalies that text summaries drop. We ask whether charts can serve as internal epistemic actions inside a data agent's reasoning loop, and how to teach an agent to use them. We first build , a suite of 200 controlled tasks under four matched interface conditions, and find a persistent gap between what agents can read and what they choose to use. Agents score as well from self-generated charts as from numeric output, yet plotting access alone does not translate into consistent use. Task slices show that charts lead on temporal, pattern, and anomaly reasoning, while text leads on precise reading and conditional verification. We then propose , which distils two families of experience from conditioned trajectory rollouts. One family records how to select and verify analytical operations, and the other records when to invoke text, charts, or both. A trajectory judge scores outcomes and labels intermediate steps, and a validation gate admits only experience that improves held-out performance. On three held-out benchmarks with four LLM backbones, raises analytical accuracy and visual evidence quality over matched baselines, and backbones that never drew charts begin using them, all without touching model parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
data analysis, chart evidence, experience-drive

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wxwaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether charts can serve as intermediate epistemic actions in agentic data analysis rather than merely as final visualization artifacts. The authors first introduce ProbeBench, a controlled benchmark of 200 analytical tasks, revealing a gap between agents' ability to understand charts and their tendency to use them during reasoning. The study also highlights the complementary roles of textual and visual representations across different analytical tasks.

Based on these findings, the authors propose ChartMind, an experience-driven framework that distills analytical and representation-selection knowledge from diverse, conditioned agent trajectories. The resulting experience package is iteratively refined through a validation gate and reused as external memory without updating model parameters.

Experiments across three held-out benchmarks and four LLM backbones demonstrate consistent improvements in analytical accuracy and visualization quality. Ablation studies further examine the contributions of conditioned rollouts, validation gating, and the two types of experience.

Critical strengths

1. Interesting problem formulation and insightful empirical observations. The paper presents an interesting perspective on treating charts as intermediate reasoning actions rather than final presentation artifacts. The controlled study on ProbeBench reveals a meaningful capability–use gap and provides useful evidence of the complementary strengths of textual and visual representations.

2. Consistent improvements across diverse settings. ChartMind demonstrates improvements across three downstream benchmarks and four LLM backbones without parameter updates. The ablation studies also provide evidence for the effectiveness of conditioned rollouts, validation gating, and the complementary roles of analytical and representation experience.

Critical weaknesses

1. Limited methodological novelty and insufficient comparison with simpler alternatives. The proposed framework primarily combines established techniques, including conditioned trajectory generation, LLM-based experience distillation, external memory, and validation-based refinement. While their integration is reasonable, the methodological contribution appears relatively incremental. More importantly, the paper does not establish whether such a complex framework is necessary compared with simpler alternatives, such as rule-based representation selection, direct prompting with visualization guidelines, or lightweight LLM-based routing.

2. Insufficient evidence that charts genuinely function as epistemic actions. The central claim is that charts actively contribute to intermediate reasoning rather than merely improving the final presentation. However, the current evaluation mainly measures final task performance, visualization quality, and chart-generation frequency. These metrics do not directly establish whether agents actually use visual observations to revise their subsequent analytical decisions. Furthermore, the ablation results suggest that general analytical experience accounts for a substantial portion of the performance improvement, making it difficult to isolate the contribution of chart-based reasoning. This weakens the empirical support for the paper's central claim.

Questions for authors

Q1. How can the authors demonstrate that agents actually use intermediate chart observations to guide subsequent reasoning? For example, would removing or masking chart observations while preserving chart-generation actions significantly affect analytical performance?

Q2. How does ChartMind compare with simpler representation-selection strategies, such as predefined rules based on task types, direct prompting with visualization guidelines, or LLM-based selection between textual and visual representations? Such comparisons would help establish the necessity of experience-driven distillation.

Q3. The ablation results show that analysis experience alone accounts for a substantial portion of the overall improvement. To what extent can the gains of ChartMind be attributed specifically to learning when and how to use charts, rather than to general improvements in analytical reasoning?

Q4. How reliable are the step-level attribution labels generated by the trajectory judge? Is there any human validation or agreement analysis supporting its ability to distinguish helpful representation choices from helpful analytical operations?

Q5. ChartMind increases the number of tool-call rounds in the reported trajectory analysis. How do the improvements compare with the additional inference costs, including token consumption, tool executions, and latency? Do the gains remain meaningful under matched computational budgets?

Q6. How sensitive is the learned experience package to the backbone used for distillation? Since the final package is constructed using a single backbone, would distilling experience from a different model produce similar transfer performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer: wxwa

Thanks for your careful and valuable comments. We will refine the paper following your suggestions. We provide detailed responses to your concerns below.

Q1: Evidence that the agent uses intermediate chart observations to guide subsequent reasoning

A1: Thank you for this important question. Following your suggestion, we conduct chart-masking experiments on DAComp-DA with GPT-5.6-Luna:

  • ChartMind, default. The agent can view charts during both analysis and final answering.
  • ChartMind, chart masked. We preserve the original trajectory and chart-generation actions, mask the charts, and regenerate only the final answer.
  • ChartMind, blind. We regenerate the full trajectory, the agent can plot charts but cannot view them.
SetupAcc.Vis.Overall
Baseline46.617.6654.32
ChartMind, default51.1827.6159.78
– chart masked46.7726.4359.25
ChartMind, blind46.8116.3357.30

Compared with default, chart masked keeps the same trajectory but removes chart access when generating the final answer. Acc. drops from 51.18 to 46.77, close to the baseline (46.61), indicating that chart observations carry evidence required for correct conclusions that is not fully captured in the textual trajectory. Compared with chart masked, blind further removes chart access during analysis. This lowers Vis. (26.43 to 16.33) and Overall (59.25 to 57.30), while Acc. remains similar (46.81). Thus, viewing charts during analysis mainly shapes how the agent selects and constructs subsequent visual evidence. In summary, charts provide critical evidence for analytical reasoning and decision-making.

We further conduct a trajectory attribution analysis on the baseline and ChartMind default. We measure the numbers of text and chart observations, as well as the numbers of observations that support the final answer.

MethodText obs.Text helpfulChart obs.Chart helpful
Baseline5673421311
ChartMind5523356137

ChartMind adds 48 chart observations and 26 helpful chart observations. Helpful chart evidence increases from 11 to 37, a 3.36× increase, while both text measures remain similar. Together with the masking results, this finding shows that experience learning enables agents to use visual evidence to improve downstream analysis.

Q2: Comparison with simple representation-selection strategies

A2: Following your suggestion, we compare learned representation experience with three simpler alternatives. For predefined rules, we manually define a set of rules and directly prompt the Agent. For the LLM-based method, we use gpt-5.6-sol as an expert to generate task-specific experience for each trajectory, and prompt the main agent. The experimental results are as follows:

MethodAcc.Vis.Overall
Baseline46.617.6654.32
Predefined rules48.1019.3257.10
LLM-based50.3017.9157.43
ChartMind51.1827.6159.78

ChartMind outperforms all comparison methods, achieving an Overall score of 59.78. Predefined rules and the LLM-based method improve the baseline Overall score to 57.10 and 57.43, respectively. These results demonstrate the effectiveness of our method.

Q3: To what extent are ChartMind’s gains chart-specific rather than general analytical reasoning improvements?

A3: Thank you for raising this point. Thank you for raising this point. We agree that, on top of analysis experience, the marginal Acc. gain of representation experience is small (+0.87; Overall 58.74 to 59.78). However, this marginal value understates its contribution. First, representation experience improves accuracy on its own: the representation-only setting raises Acc. from 46.61 to 49.31 (+2.70), comparable to analysis-only (+3.70). Second, the two types of experience overlap substantially: the sum of their individual gains (6.40) exceeds the combined gain (4.57), so the marginal contribution depends on the order of addition. If representation experience is added first, the marginal Acc. gain of analysis experience is also only +1.87. This overlap is expected, as both are distilled from the same conditioned text-preferred and chart-preferred trajectories. Third, the distinct contribution of representation experience lies mainly in visual evidence: on top of analysis experience, it raises Vis. from 10.43 to 27.61, whereas analysis experience alone improves Vis. by only 2.77. We will clarify this in the final version.

Q4: Concerns about the reliability and discriminative ability of step-level attribution

A4: Step-level attribution involves reflecting on the usefulness of each step in a rollout trajectory. Each step is a single code execution that may include both analysis and representation operations. Following your suggestion, we manually annotate 20 sampled rollout trajectories and assess human–LLM agreement. The results are shown below:

Human label / LLM judgeHelpfulIneffectiveHarmfulClass agreement
Helpful470394.0%
Ineffective020100.0%
Harmful211076.9%
Overall accuracy———90.8%

The LLM judge and human annotations agree on 59 out of 65 steps, yielding an overall accuracy of 90.8%. Conditioned on the human label, the class-wise agreement rates for Helpful, Ineffective, and Harmful are 94.0%, 100.0%, and 76.9%, respectively. These results strongly support the reliability of the step-level attribution labels.

Q5: Comparison under matched computational budgets

A5: We align ChartMind's inference costs with those of each corresponding baseline trajectory. The results are shown below:

We provide additional experimental results with ChartMind’s inference budget matched to each corresponding baseline trajectory. The results are shown below:

SettingAcc.Vis.OverallMean input tokensMean output tokensMean tool calls
Baseline46.617.6654.3273,3399,3128.44
ChartMind51.1827.6159.78122,4519,73312.85
ChartMind, budget matched50.4020.2058.8075,2428,2078.74

Under approximately matched budgets, ChartMind achieves an Overall score of 58.80, exceeding the baseline by 4.48 points and retaining 82% of its full-budget gain. It uses 2.6% more input tokens, 11.9% fewer output tokens, and 3.6% more tool calls than the baseline. Thus, the gains remain meaningful at similar inference costs.

Q6: How sensitive is the learned experience to the LLM backbone used for distillation?

A6: We conduct an ablation study using different LLM backbones for distillation. The results are shown below:

Builder backboneAcc.Vis.Overall
Baseline46.617.6654.32
Claude Sonnet 549.9321.0059.41
DeepSeek V4.1 Flash51.8525.7160.89
GPT-5.6 Luna48.8625.0559.28
GPT-5.6 Sol51.1827.6159.78

ChartMind significantly outperforms the baseline across all four builder models. DeepSeek V4.1 Flash achieves the highest Overall score of 60.89, while GPT-5.6 Luna yields the lowest score of 59.28. The highest and lowest scores differ by 1.61 points. These results show that ChartMind effectively improves performance across different backbone models.

Thanks again for your important questions. Hope the additional evidence could addresses your concerns, and we would appreciate it if you could reassess our work and increase our rating.

I thank the authors for their response. My concerns have been largely addressed. Although I do not consider the methodological novelty of this paper to be very high, the insights and findings are valuable. I have adjusted my score to 3.

Thank you for your thoughtful feedback and for reassessing our work. We are glad that our response has largely addressed your concerns, and we appreciate your recognition of the value of our insights and findings.

Thank you again for your time and effort throughout the review process.

Reviewer_C4zsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether charts should function as internal epistemic actions (intermediate evidence) inside a ReAct data-analysis agent's loop, rather than as post-hoc report artifacts.

Critical strengths
  1. The question is fresh and well motivated. Whether self-generated charts can serve as intermediate evidence in an agent's own reasoning loop is understudied relative to chart generation (for users) and chart understanding (of given charts). The capability–use distinction (F1 vs. F2) is a sharp, empirically grounded way to frame the problem: agents can read charts they draw, yet most backbones do not draw them when free to choose.
  2. The probe design is clean at the interface level. Holding the backbone, context, tool budget, and one-Bash-tool environment fixed across four matched conditions (§2.1) is the right controlled comparison, and the task-paired slice analysis with clustered bootstrap and randomization tests is the right statistical framing in principle.
  3. The conditioned-rollout idea is sensible. Forcing text-preferred and chart-preferred trajectories (with chart-family exclusion) on the same task creates paired contrasts so that a text-habit backbone still generates chart-led evidence; without this, distillation would simply reproduce the backbone's prior habit. The 200-task × 5-rollout bundle and type-exclusion rule are a practical, low-cost design.
  4. Good engineering hygiene in places. Train/validation/test split with all trajectories of a task in one split; downstream benchmarks held out of distillation; a single package built with one backbone and transferred to all four backbones (cross-model transfer); freezing before external evaluation; explicit token accounting (Tables 7–9), including the downstream overhead; and full release of the experience documents (Appendix D).
  5. The validation gate is the right conceptual control against builder overwrites, and the two-family storage (analysis vs. representation) is auditable rather than a black-box memory.
Critical weaknesses
  1. The evaluation chain is circular: the same LLM judge supplies training labels, the validation gate, and the main downstream results — and the judge is never validated.
  2. The central causal mechanism is not demonstrated — and the paper's own Figure 5 shows it. The title-level claim is that charts act as in-loop epistemic actions that improve reasoning. The required evidence is that in-loop chart use rises and mediates the accuracy gain. Instead: §4.4 and Figure 5 state that the analysis-phase chart share (excluding the final two report calls) "stays low and does not consistently increase across backbones," and explicitly warns that raw chart counts overstate in-loop chart reasoning (the model may just plot near the end). So the model changes behavior only partly. The accuracy gains are mostly attributable to generic analysis rules, not representation knowledge: baseline Acc 46.61 → E_ana-only 50.31 → full 51.18 (the representation family contributes only 0.87 of the 4.57-point accuracy gain on Luna; Table 6). E_ana-only Overall 58.74 is within 1.04 of the full 59.78. No mediation analysis links chart usage to correctness (e.g., per-task: does the presence of an in-loop chart on this task predict accuracy improvement over baseline, controlling for extra rounds/tokens?).
  3. Missing content-matched control — the baseline is not fair to the method. The CHARTMIND agent receives an extra catalog metadata block, a kb_read tool, an extra system-prompt section, and 20–67% more input tokens per task (Table 9). The baseline receives nothing comparable. The natural control — a generic, static data-analysis/visualization guideline package of equal token budget and identical retrieval interface (without conditioned-rollout distillation or modality lessons) — is absent. The "analysis-only" and "representation-only" ablations are both products of the same expensive distillation pipeline; they do not control for priming, extra context, or generic advice. Without this control, the gains are compatible with "give the agent a well-written analyst checklist."
Questions for authors

How sensitive are the results to the builder backbone (package built with Sol only) and to rendering quality (resolution, chart families available)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer: C4zs

Q1: Concerns that the same LLM judge makes the evaluation chain circular

A1: Thank you for your question. We address this concern from two perspectives.

First, there is no circularity in the downstream evaluation. We strictly follow the official evaluation protocols of the main downstream benchmarks. The LLM judges for DAComp-DA, DDR-Bench, and DSBench are gemini-2.5-flash, gpt-5-mini, and gpt-4o, respectively; all differ from the models used by ChartMind. We will clarify this in the final version.

Second, during experience distillation, the ChartMind default setting uses gpt-5.6-sol to provide training signals and perform validation gating. These processes operate on different trajectories with independent contexts. We further use different models for validation gating. The results are as follows:

Gate modelAcc.Vis.Overall
Baseline46.617.6654.32
GPT-5.6 Sol (default)51.1827.6159.78
GPT-5.6 Luna50.5824.2059.29
Claude Sonnet 551.3326.4059.47
DeepSeek V4.1 Flash50.3228.8060.38

All methods outperform the baseline across different gate models. DeepSeek yields the highest score of 60.38, while Luna yields the lowest score of 59.29.

Q2: Evidence that charts serve as epistemic actions and improve reasoning

A2: Thank you for raising this important question. We acknowledge that Sec. 4.4 and Fig. 5 show only that ChartMind generates charts during analysis; they do not establish that these charts contribute to the analysis. We provide additional evidence and clarify the relevant details below.

First, the original Fig. 5 normalized chart observations by all tool calls, so the additional kb_read calls lower the reported proportion of chart observations. We therefore report the proportion of chart observations among bash calls:

LLM BackboneMethodChart obs.Text obs.Chart obs. / all tool calls (old)Chart obs. / bash calls (revised)
Claude Sonnet 5Baseline1572,7494.39%4.62%
ChartMind2133,3435.54%6.24%
DeepSeek V4.1 FlashBaseline2414,4074.23%5.26%
ChartMind3044,1105.17%7.15%
GPT-5.6 LunaBaseline135671.64%1.91%
ChartMind615525.16%8.94%
GPT-5.6 SolBaseline1018958.37%10.72%
ChartMind1239517.22%12.53%

ChartMind increases both the number and proportion of chart observations across all four LLM backbones. Luna shows the largest gain, from 1.91% to 8.94%, while gpt-5.6-sol shows an increase from 10.72% to 12.53%. We will clarify this issue in the main text and update Fig. 5 using the revised metric.

Second, we conduct an intervention experiment to test whether chart observations affect the analysis:

  • ChartMind, default. The agent can view charts during both analysis and final answering.
  • ChartMind, chart masked. We preserve the original trajectory and chart-generation actions, mask the charts, and regenerate only the final answer.
  • ChartMind, blind. We regenerate the full trajectory, the agent can plot charts but cannot view them.

We conduct this experiment on DAComp-DA with GPT-5.6 Luna. The results are as follows:

SetupAcc.Vis.Overall
Baseline46.617.6654.32
ChartMind, default51.1827.6159.78
– chart masked46.7726.4359.25
ChartMind, blind46.8116.3357.30

Compared with default, chart masked keeps the same trajectory but removes chart access when generating the final answer. Acc. drops from 51.18 to 46.77, close to the baseline (46.61), indicating that chart observations carry evidence required for correct conclusions that is not fully captured in the textual trajectory. Compared with chart masked, blind further removes chart access during analysis. This lowers Vis. (26.43 to 16.33) and Overall (59.25 to 57.30), while Acc. remains similar (46.81). Thus, viewing charts during analysis mainly shapes how the agent selects and constructs subsequent visual evidence. In summary, charts provide critical evidence for analytical reasoning and decision-making.

Q3: Concerns about performance gains from generic analysis rules rather than representation experiences

A3: Thank you for raising this point. We agree that, on top of analysis experience, the marginal Acc. gain of representation experience is small (+0.87; Overall 58.74 to 59.78). However, this marginal value understates its contribution. First, representation experience improves accuracy on its own: the representation-only setting raises Acc. from 46.61 to 49.31 (+2.70), comparable to analysis-only (+3.70). Second, the two types of experience overlap substantially: the sum of their individual gains (6.40) exceeds the combined gain (4.57), so the marginal contribution depends on the order of addition. If representation experience is added first, the marginal Acc. gain of analysis experience is also only +1.87. This overlap is expected, as both are distilled from the same conditioned text-preferred and chart-preferred trajectories. Third, the distinct contribution of representation experience lies mainly in visual evidence: on top of analysis experience, it raises Vis. from 10.43 to 27.61, whereas analysis experience alone improves Vis. by only 2.77. We will clarify this in the final version.

Q4: Experimental results with content-matched control

A4: Following your suggestion, we manually create a generic, static data-analysis and visualization guideline package and conduct a content-matched control experiment. We align the token budget of each benchmark trajectory with that of its corresponding baseline trajectory. After every tool call, the agent receives a brief message reporting its current token usage and remaining budget and prompting it to finalize the answer before the budget is exhausted. The results are shown below:

SettingAcc.Vis.OverallMean input tokensMean output tokens
Baseline46.617.6654.3273,3399,312
– Natural-controlled47.2612.4056.6375,9668,615
– ChartMind-controlled50.4020.2058.8075,2428,207
ChartMind51.1827.6159.78122,4519,733

With input token usage comparable to the baseline, ChartMind-controlled retains 82.1% of the Overall improvement achieved by full ChartMind. It also outperforms the natural control by 2.17 in Overall (58.80 vs. 56.63), supporting ChartMind’s advantage beyond additional context or generic guidance.

Q5: Ablation on the ChartMind builder backbone

A5: We conduct an ablation study using different LLMs as the ChartMind builder. The results are shown below.

Builder backboneAcc.Vis.Overall
Baseline46.617.6654.32
Claude Sonnet 549.9321.0059.41
DeepSeek V4.1 Flash51.8525.7160.89
GPT-5.6 Luna48.8625.0559.28
GPT-5.6 Sol51.1827.6159.78

ChartMind significantly outperforms the baseline across all four builder models. DeepSeek V4.1 Flash achieves the highest Overall score of 60.89, while GPT-5.6 Luna yields the lowest score of 59.28. The highest and lowest scores differ by 1.61 points. These results show that ChartMind effectively improves performance across different backbone models.

Q6: Ablation on chart rendering and available chart families

A6: We conduct two ablation studies to evaluate the effects of chart rendering quality and the range of available chart types in the chart environment.

Chart rendering quality. By default, ChartMind allows the agent to determine the chart resolution, typically 1280×7201280 \times 720. We compare this default configuration with low-resolution (below 320×240320 \times 240) and high-resolution (above 1920×12801920 \times 1280) settings. The results are shown below:

Rendering qualityAcc.Vis.Overall
ChartMind51.1827.6159.78
– low46.0011.5256.71
– high50.7926.1559.63

Lower resolution reduces the overall score from 59.78 to 56.71 and reduces the Vis. score from 27.61 to 11.52. Higher resolution did not improve analysis performance, and overall score decreased slightly from 59.78 to 59.63. These results suggest that low resolution harms visual performance, but higher resolution is not necessarily better. The default resolution selected by Agent is sufficient for analysis.

Available chart families. By default, ChartMind applies more than 20 chart types for analysis. We compare this default configuration with two alternatives: a minimal set (bar, line, and pie) and a common set of nine chart types (including bar, line, pie, waterfall, violin, and others). The results are shown below:

Chart familiesAcc.Vis.Overall
ChartMind51.1827.6159.78
– minimal (line, bar, pie)45.4620.6757.59
– common chart types47.5526.3058.07

Restricting chart types to the minimal and common sets resulted in overall scores of 57.59 and 58.07, respectively. In comparison, ChartMind, which allows all chart types, scored 59.78. These results suggest that a wider range of chart types helps improve performance.

Thanks again for your careful and valuable comment. Hope our response can addresses your concerns. We will revise the paper following your suggestions. We would appreciate it if you could reassess our work and increase the rating.

Reviewer_FcohOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how data analysis agents select text, tables, and charts as intermediate evidence during multi-step analysis. The authors first introduce ProbeBench, a benchmark comprising 200 tasks, and compare the effectiveness of different evidence representations across four tool interfaces. Their findings show that charts can support analysis, but access to plotting tools does not necessarily prompt agents to use them proactively. Text and charts also offer complementary benefits across different task types. Authors then propose ChartMind, which generates execution trajectories with different representational preferences for the same task. An LLM scores the final answers and evaluates key steps, after which a Builder distills analytical and representational lessons from these trajectories. Candidate lessons are retained only after passing a validation gate and are ultimately compiled into a fixed external document that downstream agents can consult as needed. Using a single shared collection of lessons, the experiments demonstrate consistent improvements in the Overall metric across three downstream benchmarks and four base models. The paper’s main contributions lie in its problem formulation, its process for constructing reusable lessons, and its demonstration of cross-task transfer, rather than in model parameter training.

Critical strengths
  1. The research question is both valuable and interesting, addressing a specific and important decision in data analysis agents. By treating how intermediate results should be examined as part of the analytical strategy, the paper tackles a problem with clear practical significance. Data analysis involves not only selecting computational operations but also choosing how to represent evidence. Charts can help reveal trends, anomalies, and distributional structure, whereas tables or text may be better suited to examining precise values and verifying conditions. Studying how agents make these choices during analysis is more informative than simply documenting whether they have access to plotting tools. ProbeBench separately examines whether agents can extract useful information from charts and whether they proactively choose to use charts during unconstrained analysis. This distinction also helps clarify the motivation for the study.
  2. The learned lessons are inspectable, and costs are disclosed. Appendix D provides concrete examples of these lessons, including defining metrics precisely, breaking down the sources contributing to averages, preferring text for simple results, and verifying results after generating charts. These examples help clarify what the system has actually learned. The authors also report token costs for both offline construction and downstream execution. This transparency deserves credit, and the value of the disclosure itself should not be discounted simply because the costs are relatively high.
  3. The experiments include several meaningful controls, and the overall design is fairly comprehensive. ProbeBench holds the base agent, initial context, and budget settings constant across interface conditions. The main experiments include matched baselines using the same base model and execution framework. Lesson construction is conducted separately from the three downstream benchmarks. Tables 5–6 present ablations examining one-shot distillation, conditioned trajectories, validation gating, and the two types of lessons. Together, these design choices provide substantive empirical support for the paper.
Critical weaknesses
  1. A gap remains between the central claim and the available evidence: an effective lesson library does not, by itself, validate the role of intermediate visual feedback. The title and main narrative emphasize charts as epistemic actions that help agents form judgments and determine their next steps during analysis. However, the most direct evidence currently consists primarily of final scores and plotting statistics. The experiments do not establish whether charts provide critical information during intermediate steps or improve subsequent decisions. Figure 5 has a similar limitation. The authors measure plotting during analysis using the proportion of image-producing Bash calls after excluding the final two Bash calls. This metric demonstrates that images are generated during intermediate stages, but it does not establish whether agents use information from those images to change their subsequent analysis.
  2. The distinction from existing methods for learning from experience is insufficiently developed, and strong baselines are missing for assessing the value of the specialized design. ChartMind’s overall pipeline consists of collecting trajectories, evaluating performance, distilling natural-language lessons, iteratively revising a document, filtering candidates through validation, and making the resulting document available to agents. Closely related approaches already exist. For example, ACE (Agentic Context Engineering) treats context as a continuously updated playbook, accumulating and revising strategies through generation, reflection, and curation. This does not imply that ChartMind lacks novelty. However, the current related-work discussion focuses mainly on data analysis agents, chart generation and understanding, and representation selection, while giving insufficient attention to the directly relevant literature on learning from experience and context optimization. I would like to see a targeted comparison with a representative general-purpose method for learning from experience, using the same construction data, model, and interface for accessing the lessons.
Questions for authors
  1. How much additional benefit do the learned representational lessons provide over an agent using the same analytical lessons with simple, manually specified representation rules? What distinguishes ChartMind’s specialized design from a representative general-purpose method for learning from experience?
  2. Could the authors design experiments to establish whether charts provide critical information during analysis and lead agents to change key decisions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer: Fcoh

Thanks for your positive assessment and thoughtful comments. We response to your conserns below.

Q1: Evidence that intermediate charts inform subsequent analytical decisions

A1: Thank you for this important question. We agree that Sec. 4.4 shows only that ChartMind generates charts during analysis, it does not establish whether these charts affect subsequent decisions. To address this concern, we design the following experiments:

  • ChartMind, default. The agent can view charts during both analysis and final answering.
  • ChartMind, chart masked. We preserve the original trajectory and chart-generation actions, mask the charts, and regenerate only the final answer.
  • ChartMind, blind. We regenerate the full trajectory, the agent can plot charts but cannot view them.

We conduct these experiments on the DAComp-DA benchmark with the gpt-5.6-luna model. The results are shown below:

SetupAcc.Vis.Overall
Baseline46.617.6654.32
ChartMind, default51.1827.6159.78
– chart masked46.7726.4359.25
ChartMind, blind46.8116.3357.30

Compared with default, chart masked keeps the same trajectory but removes chart access when generating the final answer. Acc. drops from 51.18 to 46.77, close to the baseline (46.61), indicating that chart observations carry evidence required for correct conclusions that is not fully captured in the textual trajectory. Compared with chart masked, blind further removes chart access during analysis. This lowers Vis. (26.43 to 16.33) and Overall (59.25 to 57.30), while Acc. remains similar (46.81). Thus, viewing charts during analysis mainly shapes how the agent selects and constructs subsequent visual evidence. In summary, charts provide critical evidence for analytical reasoning and decision-making.

We further conduct a trajectory attribution analysis on the baseline and ChartMind default. We measure the numbers of text and chart observations, as well as the numbers of observations that support the final answer.

MethodText obs.Text helpfulChart obs.Chart helpful
Baseline5673421311
ChartMind5523356137

ChartMind increases the number of chart observations to 61, 37 of which directly support the final conclusion. Meanwhile, the number of text observations remains largely unchanged. This indicates that ChartMind's improvement does not come from additional textual evidence, but from its effective use of chart-based evidence. These results further show that charts provide valuable evidence during analysis and positively affect the final conclusions.

Q2: Comparison with general-purpose experience-learning methods, e.g., ACE

A2: ChartMind aims to learn from experience and enable agents to use charts as epistemic actions during data analysis. Unlike ACE's generation-reflection-curation loop, ChartMind employs a build-and-gate process to distill experience from rich trajectories collected through conditioned rollouts.

Following your suggestion, we further compare ChartMind with ACE [R1] using the same distillation model gpt-5.6-sol and seed probe trajectories. The results are shown below:

MethodAcc.Vis.Overall
Baseline46.617.6654.32
ACE47.084.2954.31
ChartMind51.1827.6159.78

ChartMind outperforms ACE on Acc., Vis., and Overall by 4.10, 23.32, and 5.47 points, respectively. ACE slightly improves Acc. but reduces Vis. This may be because ACE neither generates trajectories with diverse charts nor learns from representational experience. These results indicate that ChartMind learns representational experience more effectively and achieves better performance.

[R1] Zhang, Qizheng, et al. "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models." International Conference on Learning Representations, 2026.

Q3: Learned vs. hand-crafted representation rules with the same analytical lessons

A3: We manually create 10 representation rules, each briefly specifying when and how to use text, tables, or charts for data presentation, for example:

  • Use direct text for exact numerical comparisons ...
  • Use a table for small datasets and detailed values ...
  • Use a line chart for long sequences or temporal trends ...

We evaluate them on DAComp-DA with gpt-5.6-luna. The results are as follows:

MethodAcc.Vis.Overall
Baseline46.617.6654.32
Analytical lessons50.3110.4358.74
+ Manual Rules50.1612.2957.40
+ Learned Experience (ChartMind)51.1827.6159.78

Learned experience outperforms manual rules across all metrics. Manual rules achieve an Overall score of 57.40, compared with 58.74 for analysis lessons alone, likely because the manual rules induce inappropriate data representations or conflict with the analysis lessons. These results demonstrate the advantage of learned experience.

Thank you again for your appreciation of our work and valuable comments. Hope our responses answer your questions well.

Thank you again for your thoughtful feedback and positive assessment of our work.

As the discussion period is coming to a close, we would appreciate any follow-up thoughts on our rebuttal, particularly whether any questions or concerns remain.

We would be happy to provide further clarification if needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the AC and all reviewers (Fcoh, C4zs, and wxwa) for their careful evaluation.

All three reviewers recognized the value of our research question. Their reviews also highlighted the capability–use gap, the controlled experimental design, and the consistent performance improvements.

In response to their comments, we provided additional evidence on the role of chart observations, comparisons with alternative methods, and the robustness of our approach. We will incorporate these additions and further refine the relevant sections in the final version. Currently, both Reviewers wxwa and Fcoh rate the paper 3 (Weak acceptance).

We again thank the AC and all reviewers for their time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.