acceptodds
Under review as a conference paper at ICLR 2027

OmniFigure: Unifying Representation for Data Visualization

Abstract

Reusing a visualization example requires understanding whether its analysis fits the new data. Similar-looking tables may support different analyses, while changes in storage layout can obscure useful examples. To address both difficulties, retrieval needs a shared representation of what an analysis requires and what the new data support, independent of their storage layout. We therefore develop OmniFigure, combining OmniGrammar with a library of visualization examples. OmniGrammar matches case requirements to the observation structure of new data; retrieved figures and programs guide models and coding agents. To evaluate this reuse, we introduce OmniFigureBench, covering retrieval under representation changes and relation loss, together with intent-aligned generation. Experiments show improved recovery of applicable examples and rejection of incompatible ones, with generation gains for most systems on MatPlotBench, PlotCraft, and OmniFigureBench. Our code and data are available at https://anonymous.4open.science/r/OmniFigure-7CEA/ and https://anonymous.4open.science/r/OmniFigure-Bench-EF4E/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Data Visualization, Visualization Code Generation, Structured Representations, External Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dwztOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OmniFigure, a retrieval framework for reusing visualization examples by matching what an analysis requires to the observation structure supported by new data, independently of its storage layout. Its core representation, OmniGrammar, describes visualization cases and new tasks in a shared schema and retrieves compatible figures and plotting programs as references for existing models and coding agents. The paper also introduces OmniFigureBench, which evaluates representation consistency, sensitivity to relation loss, and intent-aligned generation. Experiments report improved recovery of applicable examples, rejection of incompatible ones, and downstream generation gains for most evaluated systems on MatPlotBench, PlotCraft, and OmniFigureBench. OmniFigure does not introduce a new visualization generator; instead, it augments existing generators with structurally compatible examples.

Critical strengths
  1. The paper addresses a meaningful limitation of similarity-based visualization retrieval. OmniGrammar explicitly represents observation structure and shared identity, allowing the system to preserve valid reuse across storage-layout changes while rejecting examples whose required data relations are absent. This is a useful and well-motivated formulation of visualization example reuse.

  2. The work is systematic and substantial in scope. It combines a complete retrieval pipeline, a 1,010-case benchmark covering representation consistency, relation sensitivity, and intent-aligned generation, and evaluations across multiple general models, visualization models, and agents. The controlled representation and component ablations further help identify which structural information contributes to retrieval performance.

Critical weaknesses
  1. The independent contribution of OmniGrammar is not convincingly isolated. In Table 3, replacing the LLM retriever with OmniGrammar under the same Curated Library improves the VLM score only from 1.142 to 1.162. No paired confidence interval or significance test is provided, and the manuscript does not clearly report the sizes, construction criteria, or coverage differences of the Base and Curated libraries. Therefore, it remains unclear how much of the downstream improvement comes from OmniGrammar rather than retrieval augmentation or the larger reference library.

  2. The human evaluation is too limited to support general method-level conclusions. Only 156 outputs are rated across 54 benchmark-by-method-condition cells, corresponding to fewer than three independent outputs per cell on average. In contrast, Pass and VLM are computed over the full sets of 100, 982, and 1,010 tasks for every method condition. Having five experts rate each output improves rating reliability but does not address this limited task coverage. Moreover, all expert-rated outputs passed execution, whereas failures are included as zeros in the VLM metric, and the sampling and evaluator-blinding procedures are not described. Therefore, the Expert scores are neither directly comparable to Pass/VLM nor sufficiently representative to support the method-level differences reported in Table 1. They would be more appropriately presented separately as a limited validation of the MLLM judge, with uncertainty intervals and a clear sampling protocol.

Questions for authors
  1. Library construction. What are the exact sizes, composition, and construction criteria of the Base and Curated libraries? Is Curated a strict superset of Base, and how are duplicate or equivalent cases excluded?

  2. Uncertainty in Table 3. Are the improvements in Table 3 statistically significant, particularly the difference between LLM Retriever + Curated and OmniFigure? Please report paired confidence intervals or statistical tests.

  3. Limited human evaluation. Could the authors increase the number of expert-rated outputs? How were the 156 outputs selected, and how many outputs were evaluated for each method on each benchmark?

  4. Annotation and human review. Which LLM and prompts are used to convert visualization cases and user queries into OmniGrammar representations? How are the final benchmark annotations reviewed by humans?

3 comments

We thank the reviewer for recognizing OmniGrammar's motivation, structural representation, and the scope of OmniFigureBench. We appreciate the questions on library composition, uncertainty, human assessment, and annotation, which guided our revisions.

1. Library composition and provenance

Our goal is a reusable library of real visualization experience without redundant basic plots. Base contains 48 R/Python examples of 24 plotting forms, following plotthis. Curated adds 584 specialist cases, totaling 632, and is thus a strict superset of Base. All retrievers in the main ablation use the same frozen pool and budget.

Cases originate from Bizard, FigureYa, Python/R Graph Gallery, Vega-Lite, and publications. Each has a unique ID and a traceable code or paper link in case.json (source.name, source.url) or catalogue.csv (source_url), available in the anonymous repository.

We check provenance, executability, and analytical content, followed by manual review. Specialist cases offer identifiable differences in structure, encoding, analysis, or implementation beyond the basic plots. This retains different reusable designs without assuming every example is analytically unique. Generation pools exclude target IDs, matching URLs, identical programs, and identified overlaps. Frozen membership is recorded in reference-pools.json.

2. Independent contribution and statistical uncertainty

We appreciate the concern over the 0.020 visual-score difference in the original Table 3. Our initial evaluation emphasized appearance, whereas OmniGrammar addresses analytical applicability. Following VisEval, we retain Visual Quality (VQ) and add Scientific Adequacy (SA) for Data Fidelity, Analytical Validity, and Intent Alignment, plus an SA-gated visual score.

With the same 632-case Curated Library and generation setup, LLM retrieval and OmniGrammar achieve 72.38% vs. 82.87% SA and 9.80% vs. 1.09% major scientific errors. The paired source-cluster bootstrap SA gain is 10.50 percentage points (95% CI [7.49, 13.46]). SA-gated VLM Score increases from 1.164 to 1.422, a paired gain of 0.257 (95% CI [0.202, 0.310]).

These scientific endpoints differ from the original ungated visual metric; the new intervals do not test the former 0.020 difference. The fixed-library results demonstrate additional value in scientific completion and analytical correctness.

3. Human evaluation

We agree that sampling and reliability matter. We preselected six tasks per benchmark across difficulty levels and evaluated all nine systems in both conditions. This yields 324 assigned outputs, including 274 successfully rendered and rated; failures remain in the denominator.

Five raters, blinded to method identity and shown images in randomized order, evaluated structure, layout, text/axes, color/style, and clarity. Human visual scores are independent of the automatic scientific gate. ICC(A,5) reliability is 0.87, 0.74, and 0.84 across MatPlotBench, PlotCraft, and OmniFigureBench. Human–VLM Score Spearman correlations are 0.73, 0.69, and 0.71. As VLM Score includes scientific adequacy, these reflect association between different measures.

On OmniFigureBench, the paired Human improvement is 0.223 (95% CI [0.084, 0.398]). The other results and intervals are also reported. The human study supports visual assessment; full benchmarks underpin overall conclusions.

4. Annotation models, prompts, and review

We thank the reviewer for requesting the annotation protocol. GPT-5.6 Sol annotates library cases and generation queries through the OmniGrammar SDK. Case annotation uses data, plotting code, and figure summaries; queries use user requests and data. We release annotation-prompt.txt and validate schemas, relationships, and bindings against input data.

To examine annotation quality separately from matching, DeepSeek V4.1 Flash and GPT-5.6 Sol annotate the same T1 queries with fixed candidate contracts and matcher. H@1 is 14.46% vs. 33.09%, indicating the importance of query-structure inference for early ranking.

All released Grammars undergo manual checks of observation units, fields, relationships, bindings, and use in the source programs. T2 relation labels are established before retrieval; alternative-candidate judgments are independently reviewed. The appendix details validation and regeneration.

We thank the reviewer again. These revisions make the evidence and procedures easier to assess while clarifying our two complementary contributions: real examples supply design and implementation experience, and OmniGrammar selects examples analytically compatible with new data.

Thank you for the detailed response. The clarifications and additional results have addressed my main concerns. I hope the authors will incorporate these new experimental results, evaluation details, and implementation information into the revised paper. Based on the response, I will raise my rating from 2 to 3.

Thank you very much for revisiting our response and recognizing that the additional experiments and clarifications have addressed your main concerns. Your suggestions on library construction, statistical evidence, human evaluation, and annotation procedures have been invaluable in improving the rigor, transparency, and reproducibility of our work.

We will incorporate the new experimental results, evaluation protocols, and implementation details into the revised manuscript, as you suggested. We sincerely appreciate your thoughtful review, constructive feedback, and support for our work.

Reviewer_PfxLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OmniFigure, a retrieval-augmented approach to visualization generation. It annotates a library of plotting examples with OmniGrammar, a schema of the data structure each example requires, and retrieves only examples whose requirements the new data can satisfy. The retrieved figures and code are given to a generator as references. The paper also introduces OmniFigureBench for evaluating retrieval and generation, and reports gains over retrieval baselines and for most generators on three benchmarks.

Critical strengths

The compatibility check is precisely defined and comes with formal guarantees. Evaluating recovery and rejection jointly is a sound choice, and the paper shows that a retriever can score well on rejection alone while failing the joint test.

Critical weaknesses
  1. The need for example retrieval is not established. When the task is specified in detail, as in MatPlotBench, the gains are marginal: execution improves by only 1.56 points overall, and two generators get worse. The authors also note that simple tasks gain little. On their own benchmark, execution improves by 7.02 points, which suggests the main benefit is showing the model how to call the right plotting libraries, since the library and the test tasks come from the same sources. The paper never compares against a generator that receives detailed guidance or documentation instead of retrieved examples.

  2. OmniGrammar adds little beyond plain retrieval. A simple LLM retriever over the same library already provides most of the generation gain. OmniGrammar improves on it by only 0.02 on a 0–2 scale, tested with a single generator and no significance test. Its large retrieval advantage comes from tests built around the very distinctions OmniGrammar encodes: T1 queries already include variable identity metadata, and the layout changes highlighted in the introduction are not evaluated. The motivating harm, a plot implying individual changes that were never observed, is never measured in the generated figures.

Questions for authors

How does OmniFigure compare with giving the same generator detailed guidance or library documentation, without retrieved examples? Can you show that generators using plain retrieval actually produce unsupported analyses, and that OmniFigure reduces them?

4 comments

We thank the reviewer for the thoughtful questions about OmniFigure's motivation, evaluation, and OmniGrammar's contribution. We especially appreciate the point that execution and visual appearance alone cannot establish scientific correctness. Our original aim was to evaluate both visual presentation and faithful scientific expression, but the initial metrics emphasized appearance. Following VisEval (Chen et al., 2025), we retain Visual Quality (VQ) and add Data Fidelity, Analytical Validity, and Intent Alignment to define Scientific Adequacy (SA) and an SA-gated visual score.

Our scope: integrated scientific figures

Our focus is complex, integrated scientific figures and the reuse of their design and analytical experience. Such figures coordinate specialized components and different observation levels; their difficulty lies in preserving identities, calculations, and relationships across them. Otherwise, successful rendering can still communicate unsupported conclusions (Weissgerber et al., PLOS Biology, 2016).

MatPlotBench emphasizes specified plotting tasks. Its execution metric is already near saturation for strong agents: CoDA (ICLR 2026) reports 99% execution success on MatPlotBench, and MatPlotAgent 97% in its published evaluation. PlotCraft advances complex visualization and iterative refinement. OmniFigureBench complements both by evaluating analytical structure, scientific validity, and transfer of scientific figures to new data. Representative comparisons and actual failure trajectories appear in our supplementary reports.

Why examples beyond documentation?

We appreciate the suggestion to test documentation as an alternative and add a Document baseline. On 1,010 tasks, Document improves SA from 45.94% to 72.28%, confirming that plotting-library guidance helps.

Real examples additionally show how specialized tools, analyses, and layouts form coherent figures. Bako et al. (IEEE TVCG, 2023; 2025) establish the value of visualization examples and schema similarity in design transfer.

To test whether documentation suffices in our target setting, we add Expert tasks based on real scientific workflows, requiring specialized plotting and coordinated analyses. With identical tasks, generator, and execution budget, SA is 44.17% (ReAct), 48.33% (Document), 69.17% (LLM retrieval), 75.83% (OmniGrammar), and 78.33% (Hybrid). Document gains only 4.16 points over ReAct: although all tasks render, 61 outputs omit a requested comparison. Complete examples yield much greater scientific-task completion. We also report major error rates to distinguish completion from reliability.

OmniGrammar's independent contribution

Examples provide design and implementation experience; OmniGrammar selects examples whose analytical requirements fit the new data. Fixing the same 632-reference Curated Library, generator, tasks, and budget, OmniGrammar raises SA from 72.38% to 82.87% over LLM retrieval: a paired +10.50 points (95% CI [7.49, 13.46]). Major scientific errors fall from 9.80% to 1.09%.

Two experiments clarify why. In T2, compatible alternative P@5 rises from 55.60% to 74.80%. In the Matched Reference Experiment, references are matched on chart family, language, layout, style, and context budget while contrasting analytical compatibility. Compatible references improve SA by 24.58 points and reduce major errors by 15.42 points. This isolates the value of compatibility; the same-library comparison tests OmniGrammar's actual retrieval contribution.

A paired Expert case illustrates the risk: LLM retrieval uses clinical-stage values for the wrong predictor alongside Cox forests, while OmniGrammar preserves the intended predictor and cohort identities. Further cases are documented in our supplementary report.

Potential to scale

OmniGrammar has the potential to scale to larger visualization libraries, an original design goal. Library structures are pre-annotated; each query is encoded once before deterministic matching. Across 32–272 candidates, online time stays at 16.7–16.9 s for OmniGrammar, while LLM retrieval rises from 16.1 to 23.4 s. Offline library preparation is excluded from this comparison.

We thank the reviewer again. OmniFigure makes integrated scientific-figure experience reusable: figures and programs supply design and implementation knowledge, while OmniGrammar enables more reliable transfer through analytical compatibility matching.

We thank the authors for the detailed response and the new experiments, especially the Document baseline and the Expert tasks, which directly address my original question.

My main follow-up concerns the Document results. On the 1,010 tasks, Document reaches 72.28% SA, close to LLM retrieval at 72.38%. On the Expert tasks, it drops to 48.33%, yet its major error rate there is the lowest of all conditions at 0.83%, compared with 12.50% for OmniGrammar. This suggests that Document mostly fails by omitting requested comparisons rather than by making errors. Could you explain which comparisons are typically missed, and why examples help with them but documentation does not?

Could you also clarify what the Expert tasks are? Is this evidence based on only 120 tasks? How is Expert difficulty defined, and who constructed these tasks?

We thank the reviewer for recognizing the new Documentation baseline and Expert experiments. Your questions address two motivations of OmniFigure: whether the design judgment embodied in real scientific figures can be transferred to general-purpose models, and how relevant examples can be retrieved reliably and efficiently.

1. Why does Document omit essential analyses, and how can real examples help?

Across 120 Expert tasks, Document rendered every figure with a major scientific error rate of 0.83%, yet 61 outputs omitted required comparisons, leaving only 48.33% scientifically adequate. Such omissions typically involved multiple outcomes, measurements, or groups, or relationships that must be preserved across panels, including shared observational units and statistical correspondences. Documentation helps models use plotting libraries correctly, but often provides insufficient guidance for coordinating these analytical requirements.

We provide a side-by-side report of actual Document and OmniGrammar outputs to illustrate these differences.

Scientific figures and their programs capture researchers' choices about visual encoding, layout, analytical relationships, and domain-specific conventions. These are precisely the experiences OmniFigure seeks to transfer through retrieval. OmniGrammar selects references by analytical compatibility, enabling their application to new data. On the same Expert tasks, OmniGrammar successfully completed 91 tasks versus 58 for Document. We also report scientific error rates for all methods to assess reliability alongside task completion.

2. How are Expert tasks defined, curated, and evaluated?

Expert extends the High complexity level by requiring at least one domain-specific computation, beyond complex visual structure. These tasks often require specialized computational packages and coordination across analytical and plotting tools. We therefore evaluated them separately from the original 1,010 general plotting tasks. Reviewer 2's questions about Documentation prompted this focused evaluation.

Expert cases originate from publicly available scientific figures within community-contributed visualization resources. Each task retains its original source and unique identifier, together with an explicit analytical request, executable data, and reference program. Provenance can be checked in the anonymous repository, as explained in our response to Reviewer 3.

Difficulty levels were cross-assessed by domain experts with doctoral degrees or extensive professional experience, considering figure structure, domain-specific analysis, implementation demands, and cross-component relationships. Models and the OmniGrammar SDK assisted in constructing structured representations; human reviewers then checked each case for observational units, field identities, variable relationships, and physical bindings. The revised manuscript clarifies this process.

All 120 Expert tasks were evaluated with five methods under identical task inputs, data, generators, and execution budgets, yielding 600 outputs. Given the effort required to curate specialized scientific tasks, this provides a substantial controlled evaluation, supported by broader experiments.

On the original 1,010 tasks, Document achieved 72.28% SA, close to the standard LLM Retriever's 72.38%, confirming the value of documentation for general plotting. In a fixed-reference-pool ablation across the same 1,010 tasks and 632 references, OmniGrammar improved SA by 10.50 percentage points over the standard LLM Retriever. In the Matched Reference Experiment on 120 Expert tasks (240 paired comparisons; 480 generations), analytically compatible references improved SA by 24.58 percentage points.

Together, these results support the contributions of documentation, real-example reuse, and analytical compatibility matching. They show how OmniFigure can bring researchers' accumulated visualization expertise into generation, while OmniGrammar enables its reliable and efficient transfer to new analytical contexts.

We sincerely thank the reviewer again for the thoughtful follow-up and constructive suggestions.

We hope our latest response has clarified the two remaining questions, particularly the types of analytical comparisons omitted by the Documentation baseline and the construction, difficulty criteria, and expert validation of the expert tasks. We have also clarified the distinction between scientific adequacy and major scientific errors to provide a more complete assessment of the trade-offs.

We would be grateful to hear whether these additional clarifications have addressed your remaining concerns. If so, we would sincerely appreciate your reconsideration of the overall assessment in light of the new evidence.

We would be happy to clarify any further questions. Thank you again for your time and careful consideration.

Reviewer_UVC7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniFigure retrieves reusable visualization examples through OmniGrammar, a shared representation of the analytical requirements of a stored case and the observation structure supported by new data. Retrieved figures and programs guide subsequent generation. OmniFigureBench evaluates source recovery under representation changes, sensitivity to relation loss, and downstream intent-aligned generation.

Critical strengths
  1. The motivation identifies a concrete retrieval problem: similar fields can support incompatible analyses, while different layouts can preserve the same analytical relationships.

  2. Matching case requirements to available observation structure is a meaningful representation contribution. Controlled retrieval results and representation ablations support its value, with downstream generation results showing practical potential.

Critical weaknesses
  1. Layout invariance. The implemented T1 evaluation uses anonymized inputs. It supports naming invariance, but does not yet establish robustness to reshaping or decomposition, which motivate the broader formulation.

  2. Alternative retrieval validity. T2 tests rejection of one source case after relation loss. It does not establish compatibility of other returned examples. A small audit would clarify what source-based metrics imply about overall retrieval quality.

  3. Generation mechanism. Retrieved figures and code can help through templates as well as analytical compatibility. Separating these benefits under matched budgets would clarify the generation gains. The controlled retrieval evidence remains useful independently.

Questions for authors
  1. Which evaluated cases test substantial layout changes rather than anonymization?
  2. How often are alternative T2 retrievals compatible with the modified input?
  3. What generation benefit is attributable specifically to analytical compatibility?
3 comments

We thank the reviewer for recognizing OmniGrammar's motivation and the value of our structural retrieval experiments. Our central premise remains that a visualization example can transfer when new data support its analytical requirements, regardless of physical representation.

1. Substantive representation changes in T1

We appreciate the distinction between anonymization and actual layout changes. Anonymization reduced shortcuts through names and identifiers. We agree that the original protocol offered limited evidence for broader invariance, and have expanded T1 to 360 transformed queries from 272 source cases, across six families: long-form records (126), renaming (83), matrix storage (55), lossless decomposition (44), mixed-record layouts (32), and graph-incidence storage (20).

These transformations change physical organization while preserving the observations and relationships required by the source analysis. For example, wide-to-long reshaping changes the records and columns but retains which repeated measurements belong to the same patient. This builds on Wickham's distinction between table layout and observational units in Tidy Data (JSS, 2014). OmniGrammar represents those relationships through variables, Facts, shared identities, and bindings.

All methods use the same physical queries and 272 candidate contracts. OmniGrammar infers query structure without the target program, source ID, or relevance labels. It achieves 14.46% H@1, versus 6.25% for LLM retrieval; H@10 is comparable (91.91% vs. 92.03%). We therefore emphasize improved early source ranking under substantive representation changes, and report the full ranking results and annotation sensitivity.

2. Compatibility of alternative T2 results

We agree that rejecting a source after relation loss does not demonstrate the quality of alternative results. We added a candidate-level audit specifically to address this.

We sampled 50 T2 interventions spanning cohort splitting, marginalization, summarization, occasion-specific identity changes, and treatment confounding. The union of LLM and OmniGrammar results yielded 305 unique query–candidate pairs. Compatibility was assessed from the modified data and each candidate's analytical requirements, without retriever identity or rank, followed by independent review of all judgments.

Fixed-slot compatible P@5 is 74.80% for OmniGrammar versus 55.60% for LLM retrieval, a 19.20-point gain. Thus, OmniGrammar both excludes the original case when its required relation is absent and returns a higher proportion of valid alternatives.

We also distinguish compatibility with available data from full fulfillment of the initial user request: after an intervention, an alternative may support a narrower valid analysis without recovering the removed observations. The appendix reports the audit alongside T2 source rejection.

3. What analytical compatibility adds to generation

We agree that retrieved figures and programs themselves provide much of the generation benefit: design ideas, coordinated layouts, library usage, and implementation patterns. This is a central motivation for OmniFigure's library, consistent with Bako et al. (IEEE TVCG, 2023) on designers' use of visualization examples.

OmniGrammar adds the ability to select references whose analytical requirements fit the new data. To separate this value from general template guidance, we introduce a Matched Reference Experiment: 120 Expert tasks, 240 paired contrasts, and 480 outputs. References within each pair share language, chart family, panel layout, visual style, reference count, and context budget. The task, input, generator, and execution allowance remain fixed. The relevant difference is whether a required relationship, such as repeated-subject identity or a joint observation, exists in the data.

Compatible references improve Scientific Adequacy by 24.58 percentage points (95% source-cluster bootstrap CI [16.53, 33.07]) and reduce major scientific errors by 15.42 points (CI [−23.15, −8.68]). Analytical compatibility therefore matters even when reference form and implementation guidance are closely matched.

This experiment isolates the value of compatibility. Our same-library ablation separately measures OmniGrammar's operational contribution: under an identical 632-reference Curated Library, it improves Scientific Adequacy over LLM retrieval by 10.50 points (paired 95% CI [7.49, 13.46]). These experiments test distinct aspects of the proposed mechanism.

We thank the reviewer again. The new experiments address all three concerns while clarifying how analytical matching complements real figures and programs.

Thank you for the clarifications and the additional experiments. I’m keeping my original rating of 3.

Thank you very much for taking the time to revisit our response and consider the additional experiments. Your suggestions on representation changes, alternative retrieval compatibility, and the independent contribution of analytical compatibility to generation have helped us strengthen our experimental design and clarify the paper's central contributions.

We will ensure that these new results and their scope are accurately reflected in the revised manuscript. We sincerely appreciate your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.