acceptodds
Under review as a conference paper at ICLR 2027

Visual Data Agent: Active Visualization and Perception of Data

Abstract

LLM-based agents are increasingly used to analyze and audit structured data. However, existing agents perceive data predominantly through textual representations, such as sampled records, serialized schemas, or summary statistics. Serializing more records increases token cost and makes relevant information harder to locate in context, so agents typically inspect only a small portion of a large data source at a time. Furthermore, distributional data patterns that emerge across many records, such as a second peak in a series of values, are hard to capture. To bridge this gap, we present Visual Data Agent (VDA), an agent that actively plots data into chart-based views and captures patterns that sampled records and summary statistics can ignore. Starting with rule-based booting views, VDA actively explores data through calibrated charts paired with numerical summaries and task-specific agent notes, using coverage feedback to guide further inspection and visual memory to revisit earlier evidence. We evaluate VDA on seven public benchmarks, from ECG rhythms to satellite telemetry, and on DataPatternBench, a benchmark we construct that asks an agent to find distributional structure in real multi-table databases. On DataPatternBench, VDA improves detection F1 from 59.8% to 84.1% and discovery F1 from 43.1% to 78.0% over a strong coding agent (OpenHands) using the same base LLM, GPT-5.4. On six localization and classification benchmarks, VDA achieves an average absolute accuracy gain of 9.2% over OpenHands, averaging pooled accuracy gains across GPT-5 and GPT-5.4. In a controlled study with GPT-5.4, charts outperform numerical summaries in detection F1 (75.6% vs. 70.7%), highlighting the value of preserving distributional information beyond numerical summaries.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, data analysis, visual perception, data visualization, data auditing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s272OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Visual Data Agent (VDA), an agentic framework that enables LLM-based data agents to actively visualize and inspect structured data rather than relying solely on textual records and numerical summaries. VDA integrates schema-based booting views, calibrated chart rendering, an observation window, agent-generated notes, visual memory, and coverage feedback to support iterative data exploration.

The authors also introduce DataPatternBench, a benchmark for detecting data defects and discovering distributional patterns in real multi-table databases. Experiments across multiple LLMs, agent frameworks, and public benchmarks demonstrate improvements in data analysis performance, particularly on DataPatternBench, where VDA improves detection F1 from 59.8% to 84.1% and discovery F1 from 43.1% to 78.0% over OpenHands with GPT-5.4.

Critical strengths

1. Practical problem and effective system design. The paper addresses an important limitation of existing data agents, which often rely on sampled records and numerical summaries that may obscure distributional patterns. VDA provides a coherent framework for incorporating visual observations into iterative data exploration, with substantial improvements on data defect detection and pattern discovery tasks.

2. Comprehensive experimental evaluation. The paper evaluates multiple LLMs, agent frameworks, and benchmarks, with extensive ablations and controlled comparisons of different information representations. The introduction of DataPatternBench also provides a useful evaluation setting with verifiable ground truth for data pattern discovery.

Critical weaknesses

1. Limited methodological novelty over existing visualization-based agents. The central idea of actively generating, inspecting, and iteratively refining visualizations has already been explored in prior work, such as AgentPoirot (InsightBench) and VESTA. The proposed components, including rule-based visualization initialization, agent notes, memory, and coverage feedback, largely build upon established techniques. Although their integration is practically useful, the paper does not sufficiently distinguish its methodological contributions from these closely related approaches. Moreover, the experiments primarily compare against general-purpose agent frameworks rather than specialized visualization-based agents, making it difficult to assess the significance of the proposed framework relative to existing work.

2. Insufficient isolation of the benefits of visual perception. The reported improvements may arise from better information coverage, structured exploration, and additional tool interactions rather than visual perception itself. In particular, the comparison with complete numerical text shows comparable performance to chart-based observations, with discovery F1 slightly favoring text (76.6% vs. 75.8%). This weakens the central claim that visual perception provides an advantage beyond information completeness. Furthermore, the additional computational and token costs are not sufficiently controlled when assessing the practical benefits of the framework.

Questions for authors

Q1. Novelty Clarification. Prior works such as AgentPoirot (InsightBench) have already investigated iterative workflows in which agents generate visualizations, interpret the results, and use them to guide subsequent analysis. What are the key methodological differences between VDA and these approaches? Which components constitute genuinely new contributions beyond integrating existing visualization and agent mechanisms?

Q2. Comparison with specialized baselines. Why are specialized visualization-based agents, such as AgentPoirot, not included in the experimental comparisons? The current baselines primarily consist of general-purpose agent frameworks. Can the authors provide comparisons with these more directly related approaches, or clarify whether their mechanisms can be adapted to the evaluated tasks?

Q3. Contribution of visual perception versus information completeness. The comparison with complete numerical text suggests that visual observations do not consistently outperform equally informative textual representations. How much of VDA's improvement can be attributed specifically to visual perception, rather than improved data coverage, exploration strategies, or additional tool interactions? Could the authors provide further controlled comparisons under comparable information and computational budgets?

Q4. Effectiveness of rule-based exploration. The ablation results suggest that booting views contribute substantially to performance, while applying predefined statistical checks to the same views already achieves 77.6% discovery F1, close to VDA's 78.0%. How much does the agent's active exploration contribute beyond systematic visualization and conventional statistical analysis? Could the authors compare VDA with a stronger automated statistical analysis pipeline using the same initial views?

Q5. Computational efficiency and practical scalability. The paper reports higher cumulative token consumption for VDA than for OpenHands, and chart-based observations are more expensive than numerical summaries in the fixed-view experiments. How do the performance gains compare under strictly matched token, tool-call, or monetary budgets? Does VDA remain advantageous under resource constraints representative of practical large-scale database analysis?

8 comments
AuthorsOpenReview ↗

Response to Reviewer s272 (Part 1)

Thank you for the detailed questions, and for recognizing the practical problem, the system design and the comprehensive evaluation. We address each concern below with new experiments.

W1 & Q1. Novelty relative to AgentPoirot and VESTA. We would like to clarify that we do not claim data plotting itself as our main contribution. Instead, VDA contributes a way for the agent to perceive the database context through calibrated charts and to explore it systematically, which fundamentally differs from AgentPoirot and VESTA.

AgentPoirot generates analysis questions about a CSV dataset and writes code that draws a plot for each question. The base model never observes the plot. It reads the plotted values as text, cut to the first 50 values per axis. Any pattern beyond those values is dropped. By contrast, VDA improves this by aggregating all information into charts, and directly places charts in the model's context window. Each chart is computed from all relevant data and calibrated for better visibility, so the base model can attend to the corresponding data patterns at once.

VESTA and VDA solve different problems, based on different abilities and workflows. VESTA proposes a statistical model of the data, and each plot checks whether that model fits. Its key ability is to test and revise a hypothesis. By contrast, VDA focuses on long-horizon and context-heavy data tasks. It receives an unfamiliar database with an overwhelming number of tables and columns without a hypothesis. Its main challenge is planning how to consume this overwhelming data context. Specifically, it decomposes the exploration into many small steps and decides each step based on the plotted charts. In VESTA, a hypothesis decides what to plot. In VDA, the charts decide what to examine next, from booting views of the whole database to follow-up charts and checks with SQL or Python. Because each chart summarizes all rows of a column in a fixed number of tokens, the agent can take many such steps. Agent notes, visual memory and coverage feedback support these steps, so that the agent keeps earlier evidence and covers the whole database.

W1 & Q2. Comparison with a specialized baseline. We thank the reviewer for this suggestion. AgentPoirot can be adapted to our tasks, so we ran its official code on all 117 DataPatternBench cases with GPT-5.4. Since it analyzes one table at a time, we run it on every table and convert its insights into our findings format.

System (GPT-5.4)Tokens per caseDetection F1Discovery F1
OpenHands0.2M61.937.3
AgentPoirot1.3M58.636.6
VDA1.3M76.775.1

At a similar token cost, VDA leads AgentPoirot by 18 points on detection and 39 points on discovery (both p = 0.004). AgentPoirot draws many plots, but its model never sees them (Q1), so it performs close to OpenHands. It also does not outperform VDA on InsightBench, its own benchmark (50.2 vs 51.9).

AuthorsOpenReview ↗

Response to Reviewer s272 (Part 2)

W2 & Q3. Visual perception versus information completeness. We would like to clarify that the 76.6% vs. 75.8% comparison covers only the cases where the complete-text runs finished. On the full evaluation set, complete text reaches 64.6% discovery F1 and 77.6% detection F1, against 78.0% and 84.1% with charts (Table 11, Appendix D.1). Complete text fails 11 runs and uses about 40% more tokens, because its size grows with the data while a chart's cost stays fixed.

The paper also isolates visual perception under fixed views, where coverage, exploration and tool interactions are identical. Detection F1 is 70.7% with numerical summaries and 75.6% with charts (Section 3.3).

To better respond to the reviewer, we further ran this control on pattern discovery, the task that depends most on distributional patterns over time. We remove only the chart images from VDA and keep all other components and budgets unchanged.

System (GPT-5.4)Discovery F1
VDA without chart images50.7
VDA77.4

Removing the chart images lowers discovery F1 by 27 points (p = 0.008). Visual perception therefore contributes beyond data coverage, exploration strategies and tool interactions.

W2 & Q5. Matched budgets. We let OpenHands use as many tokens as used by VDA, on all 117 DataPatternBench cases with GPT-5.4.

System (GPT-5.4)Tokens per caseDetection F1Discovery F1
OpenHands0.2M61.937.3
OpenHands with more tokens1.3M64.846.5
VDA1.3M76.775.1

With a similar number of requests and tokens, VDA still leads by 12 points on detection (p = 0.05) and 29 points on discovery (p = 0.004). For larger databases, a chart's cost stays fixed while text grows with the rows.

Q4. Contribution of active exploration beyond systematic views and statistics. We would like to clarify that the 77.6% result is not obtained under the same setting as VDA. It is a reference that is given the definitions of the targets in advance. Its purpose is to show the score that can be reached when the target definitions are known. It is obtained by applying the exact tests and thresholds that the benchmark uses to accept each target. VDA is not given these tests and still reaches the same score.

To answer the reviewer's questions, we ran two comparisons that start from the same initial views.

Same initial viewsDiscovery F1
OpenHands analyzing these views with SQL and Python, without charts58.6
VDA without the plotting tool (Table 3)64.1
VDA77.4

The agent's active exploration contributes 13 points beyond systematic visualization, and VDA scores 19 points above statistical analysis of the same views (p = 0.016). Active exploration is therefore a major source of VDA's gain, not a minor addition to systematic views and statistics.

We will add these comparisons and controls to the paper, and revise the related work as described above.

Dear Reviewer s272,

Thank you again for your detailed review. As the discussion period ends soon, we would like to kindly check whether you have any remaining questions, as we hope our additional experiments have addressed all of your concerns.

In summary, VDA differs fundamentally from existing plotting agents, and it outperforms AgentPoirot by 18 points on detection and 39 points on discovery at a similar token cost. The gain mainly comes from our proposed visual perception component, since removing only the charts lowers the performance significantly. It is not explained by extra compute either, because OpenHands with a similar number of tokens scores 12 points lower on detection and 29 points lower on discovery. Finally, active exploration cannot be replaced by rule-based views and statistical checks. VDA scores 13 points higher on discovery than without active exploration, and 19 points higher than an agent performing statistical analysis only.

We would be glad to clarify anything further. Thank you!

Reviewer_X6jYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents the Visual Data Agent, a framework designed to augment agents with visualization capabilities for structured data analysis. Agents can create candidate charts, making data-adaptive plotting adjustments, and reasoning over the generated chart images. To evaluate VDA, the authors introduce DATAPATTERNBENCH, a benchmark constructed from Spider 2.0-lite. VDA is also evaluated on seven public benchmarks covering time-series anomaly localization, classification, insight discovery, and a boundary QA dataset. The evaluation results show that VDA improves downstream task performance while also document extensive controls on evidence representations, batching, compute overhead, and negative task boundaries.

Critical strengths
  • The motivation is clear and quite interesting.
  • The empirical study is extensive, evaluating multiple base LLMs, agent frameworks, and public benchmarks across different tasks.
Critical weaknesses
  • While the core motivation is that actively plotting data enables agents to capture patterns that textual representations miss. However, the ablation and control data reveal that the majority of the observed gains do not stem from autonomous visual exploration. Table 3 showed that the booting views is much important than the active plotting, and in some settings the active plotting degrade performance. Moreover, the instance in DATAPATTERNBENCH is paired with a display pre-selected by the booting heuristics. This raises the concern that the agent is just performing selective verification over a pre-curated view rather than autonomously discovering unknown anomalies.
Questions for authors
  1. How often does the agent invoke the plotting tool to inspect views outside the initial booting inventory during successful runs? Could VDA be more accurately characterized as an automated rule-based data profiler feeding a multimodal LLM, rather than an active plotting agent?
2 comments
AuthorsOpenReview ↗

Response to Reviewer X6jY

Thank you for the positive assessment and the precise questions. We are glad that the reviewer finds the motivation clear and interesting and the empirical study extensive. We have conducted additional experiments and addressed each question and concern below. All new results use GPT-5.4 on DataPatternBench.

W1. Role of booting views and active plotting. We agree that booting views have the largest effect in Table 3. On discovery, however, this effect comes from the model seeing the views as charts, not from the selection of views itself. Given the same views without charts, the agent falls far behind.

System (GPT-5.4)Booting viewsChart imagesPlotting toolDiscovery F1
VDA without chart imagesyesnoyes50.7
OpenHands given the booting views as a text checklistyesnono58.6
VDA without the plotting tool (Table 3)yesyesno64.1
VDAyesyesyes77.4

The first two rows are compared with a VDA run from the same period on 35 cases (p = 0.016 and p = 0.008). Booting views are therefore the first step of visual exploration, not a substitute for it. Active plotting builds on this first look. Removing it costs 13 points on discovery, and every other component also contributes on DataPatternBench (Table 3).

W1 (continued). Relation between the targets and the booting views. The booting views come from fixed rules on the schema and column types, which use no knowledge of the planted structures. The agent is not told which views contain targets. It must examine many views, decide which show a defect or pattern, and verify each one.

Q1. How often does the agent plot outside the booting inventory? The agent regularly plots views outside the booting inventory. Across 102 runs, it makes about 20 plotting calls per case, of which 5 (detection) to 7 (discovery) are outside the inventory. At least one such view appears in 71% of detection runs and 98% of discovery runs, so the agent does not stay within the pre-selected views.

Q2. Difference from a rule-based profiler. VDA starts from booting views produced by fixed rules, as a profiler would, but this is only its first step. A rule-based profiler feeding a multimodal LLM corresponds to VDA without the plotting tool, where the model sees only these rule-based views. It reaches 64.1 discovery F1, against 77.4 for full VDA (Table 3). The difference comes from the views the agent chooses itself. It plots views outside the inventory in 98% of discovery runs (Q1). We therefore see VDA as an active plotting agent that starts from a profiler-like first look.

We will add the plotting statistics and the checklist and image-removal controls to the paper.

Thank you for the detailed response and additional experiments, which adequately address my main concern regarding the agent's autonomy in visual exploration. I appreciate the authors' efforts and am satisfied with the rebuttal. Therefore, I maintain my positive rating.

Reviewer_He1vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper describes, constructs, and evaluates an agentic framework for allowing LLMs to reason about tabular timeseries data by having external tools plot this data and the agents process these plots as images. This allows for fewer tokens to be used while being able to recognize long-range dependencies in data. The process works as follows: first, a set of “booting views” is provided, that give the agent some general idea of the data, then the agent is allowed to choose among charts itself, until convergence. Throughout, the set of previous charts is saved in external visual memory, with agent-written notes being maintained in the context. The agent at each step can choose to either continue engaging with the plotting process or work with other tools.

Results indicate that, on GPT-5 models, this approach strongly outperforms other agentic frameworks. It also shows that two different agentic frameworks benefit from this tool across the board, though no improvements in cost efficiency are reported.

Critical strengths

The broad idea of leveraging an automatic plotting framework to reduce token consumption via the already-in-built visualization capability is interesting. The evaluation also seems to be fairly comprehensive, with many benchmarks, on two different models, and a few different baselines.

Critical weaknesses

The main weakness I see in this paper is a lack of cost-equal comparisons. The main purpose of this work is to improve the efficiency of agentic systems by allowing them to use charts rather than expensively traverse through the data, however, the experiments showing an improvement seem to use more tokens; as appendix B.2 says:

Across the 117 database cases with GPT-5.4, OpenHands averages 11.2 model requests and 200k cumulative API tokens per case, compared with 27.3 requests and 1342k tokens for V DA

This is a significant problem with the evaluation, as it is unclear whether giving the existing agents more leeway to use tokens while directly processing the data might have led to better results.

The other major concern is that the language for specifying graphs is not presented in sufficient detail in the paper. A formal semantics is not strictly necessary, but some full description of all the possible kinds of graphs would be helpful. Equation 2 is roughly along these lines, but then the next paragraph includes

They add views for exact duplicate counts, identifier values by record position, and temporal trends grouped by category.

which implies that equation 2 is not comprehensive. In addition, Section 2.5 says

A plotting request specifies a supported chart type, table, and columns, such as a numeric measure grouped by category.

and this request format seems distinct from the one in Equation 2 and is never specified. Given that the core of this algorithm is the agent’s interaction with these plots, the precise set of plots and language in which the framework or agent creates them is crucial to understanding the approach and deriving insights from the paper.

Questions for authors

This question would possibly be made redundant by a more specific description of the language of Section 2.5, but my question is why lock the agent into what seems like a relatively small number of potential plots? Surely a richer plot representation would improve the agent’s ability to zero in on relevant information. The automatic plots could serve as defaults when the agent does not have sufficient context to determine what information is relevant.

3 comments
AuthorsOpenReview ↗

Response to Reviewer He1v

Thank you for the constructive review, and for finding the idea interesting and the evaluation comprehensive. We have conducted additional experiments and addressed each question and concern below.

W1. Cost-equal comparisons.

We would like to clarify that we do not claim that the agent always uses fewer tokens with charts, as discussed in Appendix B.2. Instead, our key motivation is that a plotted chart provides a more effective way than text to aggregate context for data tasks, at a nearly constant token cost. This can help solve certain types of tasks that are hard to solve with text reasoning alone. We will better clarify this in the introduction.

To support this, we conducted additional experiments in which OpenHands used a similar number of reasoning tokens compared to VDA, across all 117 DataPatternBench cases with GPT-5.4.

System (GPT-5.4)Tokens per caseDetection F1Discovery F1
OpenHands0.2M61.937.3
OpenHands with more tokens1.3M64.846.5
VDA1.3M76.775.1

With a similar number of tokens, VDA still leads by 12 points on detection (p = 0.05) and 29 points on discovery (p = 0.004), so spending more tokens on text reasoning does not replace charts on these tasks.

W2. Specification of the plotting language. We thank the reviewer for this suggestion. We will add the following table of all views to Section 2.

ViewArgumentsAggregate over all rowsBootingPlotting tool
Profiletable, columnvalue distribution (histogram or top-k bars)yesyes
Trendtable, date columnrecord count per time bucketyesyes
TrendYtable, measure, date columnmean per time bucketyesyes
TrendNulltable, column, date columnmissing-value rate per time bucketyesyes
Comparetable, group column, measure, aggregateaggregate per groupnoyes
Grouped trendtable, measure, date column, group columnmean per time bucket for each groupyesno
Category sharetable, group column, date columncategory share per time bucketdiscovery onlyno
Record positiontableidentifier value against record positionyesno
Duplicate counttablenumber of exact duplicate records (text)yesno

A plotting request names a view type and its arguments, such as TrendY(table, measure, date column), on any table and column.

Q1. Why a small, typed set of plots? We share the reviewer's intuition, and we limited the set on purpose. A multimodal model reads charts imperfectly, so a flawed chart easily becomes noise rather than evidence. For example, when a column spans several orders of magnitude, a linear histogram puts almost all records into one bin and hides the pattern. We therefore keep the charts stable. Each view type is computed from all rows and calibrated by fixed rules, such as logarithmic bins in this case (Section 2.2). Removing this calibration lowers detection F1 from 85.2% to 75.0% (Table 3). A free-form plotting language would give up this control. A fixed set of views also lets coverage feedback track which views the agent has not yet inspected. The analysis itself is not limited, since the agent can still run any SQL or Python. We agree that more view types, with the automatic views as defaults, are a natural extension, and we will discuss this as future work.

We will add the token-matched comparison and the table of views to the paper, clarify the motivation in the introduction, and revise the whole paper for clarity, starting with the method overview in Section 2.

Thank you for your responses. Assuming these changes are made as specified, I have revised my score to a Weak Accept.

Thank you for raising your score and for all the constructive feedback! We promise to include the changes as specified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.