- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents OmniFigure, a retrieval framework for reusing visualization examples by matching what an analysis requires to the observation structure supported by new data, independently of its storage layout. Its core representation, OmniGrammar, describes visualization cases and new tasks in a shared schema and retrieves compatible figures and plotting programs as references for existing models and coding agents. The paper also introduces OmniFigureBench, which evaluates representation consistency, sensitivity to relation loss, and intent-aligned generation. Experiments report improved recovery of applicable examples, rejection of incompatible ones, and downstream generation gains for most evaluated systems on MatPlotBench, PlotCraft, and OmniFigureBench. OmniFigure does not introduce a new visualization generator; instead, it augments existing generators with structurally compatible examples.
Critical strengths
-
The paper addresses a meaningful limitation of similarity-based visualization retrieval. OmniGrammar explicitly represents observation structure and shared identity, allowing the system to preserve valid reuse across storage-layout changes while rejecting examples whose required data relations are absent. This is a useful and well-motivated formulation of visualization example reuse.
-
The work is systematic and substantial in scope. It combines a complete retrieval pipeline, a 1,010-case benchmark covering representation consistency, relation sensitivity, and intent-aligned generation, and evaluations across multiple general models, visualization models, and agents. The controlled representation and component ablations further help identify which structural information contributes to retrieval performance.
Critical weaknesses
-
The independent contribution of OmniGrammar is not convincingly isolated. In Table 3, replacing the LLM retriever with OmniGrammar under the same Curated Library improves the VLM score only from 1.142 to 1.162. No paired confidence interval or significance test is provided, and the manuscript does not clearly report the sizes, construction criteria, or coverage differences of the Base and Curated libraries. Therefore, it remains unclear how much of the downstream improvement comes from OmniGrammar rather than retrieval augmentation or the larger reference library.
-
The human evaluation is too limited to support general method-level conclusions. Only 156 outputs are rated across 54 benchmark-by-method-condition cells, corresponding to fewer than three independent outputs per cell on average. In contrast, Pass and VLM are computed over the full sets of 100, 982, and 1,010 tasks for every method condition. Having five experts rate each output improves rating reliability but does not address this limited task coverage. Moreover, all expert-rated outputs passed execution, whereas failures are included as zeros in the VLM metric, and the sampling and evaluator-blinding procedures are not described. Therefore, the Expert scores are neither directly comparable to Pass/VLM nor sufficiently representative to support the method-level differences reported in Table 1. They would be more appropriately presented separately as a limited validation of the MLLM judge, with uncertainty intervals and a clear sampling protocol.
Questions for authors
-
Library construction. What are the exact sizes, composition, and construction criteria of the Base and Curated libraries? Is Curated a strict superset of Base, and how are duplicate or equivalent cases excluded?
-
Uncertainty in Table 3. Are the improvements in Table 3 statistically significant, particularly the difference between LLM Retriever + Curated and OmniFigure? Please report paired confidence intervals or statistical tests.
-
Limited human evaluation. Could the authors increase the number of expert-rated outputs? How were the 156 outputs selected, and how many outputs were evaluated for each method on each benchmark?
-
Annotation and human review. Which LLM and prompts are used to convert visualization cases and user queries into OmniGrammar representations? How are the final benchmark annotations reviewed by humans?
3 comments
Thank you for the detailed response. The clarifications and additional results have addressed my main concerns. I hope the authors will incorporate these new experimental results, evaluation details, and implementation information into the revised paper. Based on the response, I will raise my rating from 2 to 3.
Thank you very much for revisiting our response and recognizing that the additional experiments and clarifications have addressed your main concerns. Your suggestions on library construction, statistical evidence, human evaluation, and annotation procedures have been invaluable in improving the rigor, transparency, and reproducibility of our work.
We will incorporate the new experimental results, evaluation protocols, and implementation details into the revised manuscript, as you suggested. We sincerely appreciate your thoughtful review, constructive feedback, and support for our work.
We thank the reviewer for recognizing OmniGrammar's motivation, structural representation, and the scope of OmniFigureBench. We appreciate the questions on library composition, uncertainty, human assessment, and annotation, which guided our revisions.
1. Library composition and provenance
Our goal is a reusable library of real visualization experience without redundant basic plots. Base contains 48 R/Python examples of 24 plotting forms, following plotthis. Curated adds 584 specialist cases, totaling 632, and is thus a strict superset of Base. All retrievers in the main ablation use the same frozen pool and budget.
Cases originate from Bizard, FigureYa, Python/R Graph Gallery, Vega-Lite, and publications. Each has a unique ID and a traceable code or paper link in
case.json(source.name,source.url) orcatalogue.csv(source_url), available in the anonymous repository.We check provenance, executability, and analytical content, followed by manual review. Specialist cases offer identifiable differences in structure, encoding, analysis, or implementation beyond the basic plots. This retains different reusable designs without assuming every example is analytically unique. Generation pools exclude target IDs, matching URLs, identical programs, and identified overlaps. Frozen membership is recorded in
reference-pools.json.2. Independent contribution and statistical uncertainty
We appreciate the concern over the 0.020 visual-score difference in the original Table 3. Our initial evaluation emphasized appearance, whereas OmniGrammar addresses analytical applicability. Following VisEval, we retain Visual Quality (VQ) and add Scientific Adequacy (SA) for Data Fidelity, Analytical Validity, and Intent Alignment, plus an SA-gated visual score.
With the same 632-case Curated Library and generation setup, LLM retrieval and OmniGrammar achieve 72.38% vs. 82.87% SA and 9.80% vs. 1.09% major scientific errors. The paired source-cluster bootstrap SA gain is 10.50 percentage points (95% CI [7.49, 13.46]). SA-gated VLM Score increases from 1.164 to 1.422, a paired gain of 0.257 (95% CI [0.202, 0.310]).
These scientific endpoints differ from the original ungated visual metric; the new intervals do not test the former 0.020 difference. The fixed-library results demonstrate additional value in scientific completion and analytical correctness.
3. Human evaluation
We agree that sampling and reliability matter. We preselected six tasks per benchmark across difficulty levels and evaluated all nine systems in both conditions. This yields 324 assigned outputs, including 274 successfully rendered and rated; failures remain in the denominator.
Five raters, blinded to method identity and shown images in randomized order, evaluated structure, layout, text/axes, color/style, and clarity. Human visual scores are independent of the automatic scientific gate. ICC(A,5) reliability is 0.87, 0.74, and 0.84 across MatPlotBench, PlotCraft, and OmniFigureBench. Human–VLM Score Spearman correlations are 0.73, 0.69, and 0.71. As VLM Score includes scientific adequacy, these reflect association between different measures.
On OmniFigureBench, the paired Human improvement is 0.223 (95% CI [0.084, 0.398]). The other results and intervals are also reported. The human study supports visual assessment; full benchmarks underpin overall conclusions.
4. Annotation models, prompts, and review
We thank the reviewer for requesting the annotation protocol. GPT-5.6 Sol annotates library cases and generation queries through the OmniGrammar SDK. Case annotation uses data, plotting code, and figure summaries; queries use user requests and data. We release
annotation-prompt.txtand validate schemas, relationships, and bindings against input data.To examine annotation quality separately from matching, DeepSeek V4.1 Flash and GPT-5.6 Sol annotate the same T1 queries with fixed candidate contracts and matcher. H@1 is 14.46% vs. 33.09%, indicating the importance of query-structure inference for early ranking.
All released Grammars undergo manual checks of observation units, fields, relationships, bindings, and use in the source programs. T2 relation labels are established before retrieval; alternative-candidate judgments are independently reviewed. The appendix details validation and regeneration.
We thank the reviewer again. These revisions make the evidence and procedures easier to assess while clarifying our two complementary contributions: real examples supply design and implementation experience, and OmniGrammar selects examples analytically compatible with new data.