acceptodds
Under review as a conference paper at ICLR 2027

ID2C: EVALUATING AND ADVANCING DOCUMENT-TO-CHART GENERATION WITH IMPLICIT REQUIREMENTS

Abstract

Generating a chart from a document requires more than locating values and writing plotting code: a system must also infer how the values should be transformed and which visual form best communicates the requested analysis. Existing text-to-chart benchmarks often make these decisions explicit, leaving their treatment by current models poorly understood. We introduce ID2C, a human-filtered benchmark of 1,000 implicit document-to-chart requests drawn from real documents and balanced across ten analytical task types. Each request leaves the construction procedure unspecified and is paired with a sample-specific rubric covering data selection, value transformation, chart-form selection, and visual encoding. Evaluation of open and proprietary models reveals persistent weaknesses in value transformation and chart-form selection, even when the generated charts are executable and visually polished. To address these failures, we construct a benchmark-independent visualization catalogue containing 199 chart forms and 744 value transformations, with each entry describing when and how it should be applied. We integrate this external knowledge source into a retrieval-augmented multi-agent harness whose form agent retrieves relevant catalogue entries before deciding the transformation and chart form. Experiments across multiple models show that the proposed harness consistently improves chart-generation quality, with the largest gains on the two planning decisions targeted by the catalogue.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Visualization, Chart Generation, LLM Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gF74OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ID2C, a 1,000-example document-to-chart benchmark intended to evaluate chart generation when the relevant data, required transformation, and visualization form are left implicit. It also proposes a retrieval-augmented multi-agent pipeline that retrieves chart-form and value-transformation knowledge from a catalogue before generating and refining the visualization.

Critical strengths
  1. The paper identifies a reasonable limitation of existing text-to-chart tasks.

  2. The benchmark is reasonably large and carefully structured.

  3. The authors conduct a human-alignment study and ablate criterion decomposition, sample-specific rubrics, and criterion-wise inference.

Critical weaknesses
  1. Although the paper motivates ID2C by contrasting it with nvBench, nvBench 2.0, VisEval, MatPlotBench, Text2Vis, and DOC2CHART, the comparison is not in depth. As a result, it is difficult to assess whether ID2C actually measures a substantially new capability, is systematically more challenging, or simply repackages existing text/document-to-visualization tasks with more implicit prompts.

  2. Despite proposing a chart-generation system, the experiments do not compare against existing visualization agents or document-to-chart systems. The primary system evaluation is an ablation among the authors' own single-shot, multi-agent, and retrieval-augmented variants. Therefore, the reported gains establish that retrieval helps their particular pipeline, but do not establish that the resulting system advances the state of the art.

  3. The proposed harness largely combines well-established components.

Questions for authors

See weaknesses

3 comments

We thank the reviewer for the feedback on benchmark differentiation, comparisons with existing systems, and the contribution of the proposed harness.

(W1) Comparison with prior benchmarks

We agree that the related-work discussion should explain the distinction at the level of inputs, requested decisions, and evaluation protocols. ID2C focuses on the combination of document-grounded evidence, implicit analytical and visual requirements, and evaluation that accommodates multiple chart forms.

To make the comparison more concrete, we provide representative requests from the benchmarks discussed in the paper, alongside an ID2C example:

BenchmarkExample requestCharacteristics
nvBench“A pie chart showing the number of faculty members for each rank.”Uses structured database tables. The request explicitly names the chart type and aggregation; evaluation matches the output against a gold VQL specification.
nvBench 2.0“Show the gross trend of comedy and action movies by year.”Introduces encoding ambiguity over structured tables, with multiple valid reference specifications. The example still explicitly states the analytical operation: examining gross trends by year.
VisEval“Show the number of male and female assistant professors.”Generates visualization code from database tables. The example specifies the aggregation; evaluation checks validity, legality against reference metadata, and readability.
MatPlotBench“Generate a series of boxplots using matplotlib and numpy libraries. The boxplots should include a basic plot, a notched plot, …”Provides detailed figure instructions with supplied or synthesized data. Evaluation compares the generated figure with a reference image, emphasizing implementation of the specified visualization.
Text2Vis“Can you visualize the overall trend in unemployment in the USA from 2000 to 2020?”Uses provided or retrieved data, with requests that name analytical operations such as trends or correlations. Evaluation uses answer matching and LLM ratings; its ambiguous-query category concerns unanswerable requests.
DOC2CHART“Assess the hotel revenues for the top 5 highest performing regions from 2021 to 2023, focusing on the trends in revenue growth and regional performance.”Uses real documents and annotator-written intents. The example specifies the measure, entities, period, and analytical focus; evaluation checks source attribution and chart-type agreement with the reference.
ID2C (ours)“Can you chart whether the job changes were concentrated in a few key industries or spread out across the board, so I can see at a glance which sectors drove the majority of the shift?”Uses document-grounded requests that express a reading goal without prescribing the transformation or chart form. The system must infer a suitable normalization and visual representation; sample-specific rubrics accommodate alternative calculations and chart forms.

ID2C explicitly evaluates whether a system infers and fulfills unstated analytical and visual requirements for a given document needed to answer the request. This capability is not separately assessed by prior works, including DOC2CHART, whose primary automated metrics focus on data attribution and reference chart-type agreement.

ID2C also extends the evaluation scope from financial reports and scientific papers to heterogeneous real-world documents across domains. Together, its document coverage and requirement-specific criteria enable systematic diagnosis of analytical-planning failures. It also provides a public resource for evaluating document-grounded chart generation, addressing a practical gap left by DOC2CHART’s unreleased dataset.

These contributions go beyond rephrasing existing visualization requests. We will clarify these distinctions in the revised paper.

(W2) Comparison with existing visualization systems

We agree that comparisons with existing visualization systems are needed to show the effectiveness of the proposed harness. We have added evaluations of Doc2Chart [1] and MatPlotAgent [2], adapting both methods to our document-to-chart setting and using the same Qwen3.6-27B backbone across all systems.

SystemTask fulfillmentData correctnessReadabilityOverall
Doc2Chart61.371.763.066.0
MatPlotAgent63.078.971.071.0
Single-shot46.375.460.060.7
Multi-agent pipeline60.977.576.570.2
Catalogue retrieval (Ours)70.577.270.273.3

Our harness outperforms both existing systems overall, with the clearest advantage in task fulfillment: it exceeds the stronger baseline, MatPlotAgent, by 7.5 points. This supports its effectiveness in addressing the analytical-planning difficulties identified by ID2C. We will include these comparisons in the revised paper.

[1] Akriti Jain, Pritika Ramu, Aparna Garimella, and Apoorv Saxena. “Doc2Chart: Intent-Driven Zero-Shot Chart Generation from Documents.” EMNLP, 2025.

[2] Zhiyu Yang et al. “MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization.” Findings of ACL, 2024.

(W3) Contribution of the proposed harness

We agree that the proposed harness can be viewed as a combination of well-established techniques. The primary contribution of this work is the problem formulation and benchmark. Our analysis in Section 5.2 identifies inferring appropriate value transformations and chart forms as major difficulties under implicit requests. Guided by this finding, we proposed a task-specific adaptation of these techniques, particularly by equipping the form-planning agent with retrieval from a catalogue of value transformations and chart forms. This design directly addresses the planning difficulties identified by our analysis, and our controlled experiments show improvements concentrated on the corresponding evaluation criteria. We will clarify this positioning in the revised paper.

Thanks for your responses. I will raise my score

Reviewer_gfKEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ID2C, a human-filtered benchmark of 1000 document-to-chart requests across 10 analytical task types. Each request specifies an analytical goal and requires LLMs to fulfill the implicit requirements. The paper proposes 7 evaluation criteria and evaluates 14 models on the benchmark. The authors identify value transformation and chart-form selection as common weaknesses. They then develop a retrieval-augmented multi-agent pipeline using a catalogue of 199 chart forms and 744 value transformations. Experiments on two open-weight models show improvements over baseline methods.

Critical strengths

The paper addresses an important problem: generating a chart requires inferring how the data should be analyzed to address the user's request, beyond extracting data and producing executable code. The document-grounded requests and sample-specific rubrics provide a useful basis for studying these decisions. The benchmark-independent catalogue of chart forms and value transformations is also a useful resource.

Critical weaknesses
  1. The evaluation criteria and their validation raise concerns about what the reported scores measure. In Figure 6, Label-encoding fidelity rewards labels "on or beside the data in the table's own units," and lists values "transformed away from the table's numbers" under partial credit. This conflicts with tasks that require percentages, growth rates, value differences, or baseline-relative indices. Presentation requires at least two additional design features for full credit, while a clear, correct chart with default styling receives partial credit. This makes benchmark scores depend on specific design choices rather than solely on whether the chart communicates the requested analysis clearly. Furthermore, the human study provides limited validation: Table 4 reports human–judge κ\kappa of 0.383 for task fulfillment and 0.247 for data correctness, with human–human κ\kappa of only 0.094 for data correctness, which is particularly low. The authors did not explain this result and its implications for using human judgments to validate the evaluator. These inconsistencies leave questions about how much the reported comparisons and diagnoses depend on the scoring protocol.

  2. The experiments do not sufficiently isolate the claimed sources of improvement. In Table 3, catalogue retrieval scores 3.1 and 1.7 overall points higher than the pipeline without retrieval. However, the retrieval version also adds a separate query-generation call, so the comparison mixes external knowledge with additional planning. There is no matched-call comparison without catalogue information. The paper attributes improvements particularly to value transformation and chart-form selection, but Table 3 reports only dimension-level averages. Section 6.2 has a related control issue: the fully specified precursor is described in Section 3.2 as containing data-location information as well as analytical and visual instructions. The experiment does not specify whether this information was removed, leaving the claimed isolation of implicit visual requirements unclear.

  3. The presentation also leaves several important details unclear. The relationship between "value transformation" and "analytical operations" is not clearly defined, particularly for tasks such as tradeoff that need not involve derived quantities. In Section 6.3, the chart labels in Figure 3 are difficult to read at the displayed size, and the exact user request is not provided, making it difficult to independently assess the case study.

Questions for authors
  1. How does the evaluator distinguish a correct transformation into percentages or growth rates from the label-fidelity penalty described in Figure 6? How sensitive are the main comparisons to removing the original-unit requirement and the additional-design requirements in Presentation?

  2. Why is human–human agreement on data correctness so low (κ\kappa = 0.094)? Please report the A/B/Tie distributions, raw agreement rates, and confidence intervals, and clarify how judgments within each human group were aggregated.

  3. Appendix C.3 states that human raters saw the source document, request, and rendered charts, whereas the judge received the request, oracle table, code, rendered chart, and criterion rubrics. How does this difference in available information affect the interpretation of Table 4?

  4. Does catalogue retrieval still improve performance when compared with a two-call planning baseline without retrieved knowledge? Please also report its effects separately on value transformation and chart-form selection.

  5. What exact text is supplied in the explicit condition of Table 5? Did it disclose the target table, rows, or columns, and can you provide a complete implicit–explicit request pair?

  6. Why are the ten tasks selected, and what is the rationale for assigning chart-form tiers to each task?

3 comments

We thank the reviewer for the detailed assessment of the scoring protocol and experimental controls. The feedback has helped us identify rubric issues, clarify the human study, and strengthen the analysis of retrieval gains.

(W1/Q1) Revising label fidelity and presentation criteria

We agree that the original label-encoding fidelity prompt could penalize transformations required by the request. We have revised it to assess whether labels correctly express the plotted quantities without penalizing valid transformations.

For presentation, we removed the requirement of at least two additional design features. The revised prompt awards full credit when the chart goes beyond a default rendering in a concrete way, without counting features. The criterion is intentionally graded relative to a plain chart: a score of 3 is reserved for charts that are easier to read than an ordinary chart with no defects, so a clear and correct chart with default styling receives 2, which serves as the default score. We also explored splitting the presentation rubric into three separate rubrics, but the decomposition did not improve evaluation reliability. Readability is a relatively intuitive property that can be assessed directly from the rendered chart, and the existing single-rubric evaluation already shows good alignment with human judgments. We therefore retain a unified presentation rubric.

We rescored the existing outputs using the revised judge and provide the updated results in the final section of this response. The system ranking remains largely unchanged from the original evaluation (Spearman's ρ = 0.996 on the overall scores).

(W1/Q2) Human agreement and its implications

We agree that the paper did not sufficiently explain the low agreement on data correctness. We also identified a reporting error in the human–human row of Table 4 (The corrected κ values are even lower, as reported below).

Our inspection of the human judgments suggests that raters sometimes treated incorrect or insufficient value transformation as a data-correctness issue, but differed in how consistently they applied this interpretation. This variation helps explain the low inter-rater agreement on data correctness and highlights the difficulty of aligning judgments on this task. As discussed in the paper, our aim is not to claim that the judge agrees with humans better than humans agree with one another, but to assess whether it provides a useful approximation of human judgments despite this variability.

We provide the full details of the human evaluation below. Two independent groups of human raters evaluated the same 106 chart pairs, assigning an A/B/Tie verdict for each axis. Individual verdicts were used directly, without aggregation across raters. The judge’s verdict was A or B depending on which chart received the higher score, and Tie if their scores were equal. Below, we report the A/B/Tie distributions, raw agreement rates, and linearly weighted Cohen’s κ, with 95% bootstrap confidence intervals.

AxisGroup 1 A/Tie/BGroup 2 A/Tie/BRaw agreement [95% CI]Linear κ [95% CI]
Task fulfillment36/28/4221/57/28.44 [.35, .54].28 [.15, .41]
Data correctness21/66/195/96/5.59 [.50, .69].05 [−.07, .18]
Readability46/10/5039/41/26.48 [.39, .58].37 [.23, .49]
Overall51/8/4739/28/39.59 [.50, .69].48 [.34, .61]

The corresponding results for LLM judge are as follows. The revised judge is used.

AxisJudge A/Tie/BRaw agreement [95% CI]Linear κ [95% CI]
Task fulfillment35/32/39.51 [.42, .60].38 [.24, .51]
Data correctness27/52/27.49 [.40, .59].24 [.09, .38]
Readability31/45/30.51 [.42, .60].40 [.27, .53]
Overall47/10/49.73 [.64, .81].60 [.46, .73]

These results support a more qualified interpretation of alignment than the original presentation of Table 4. We will report these statistics and clarify the interpretation of low human–human agreement in the revised paper.

(W2/Q4) Isolating retrieval gains from additional planning

We agree that the original comparison confounds catalogue access with an additional planning call. To address this, we evaluated the same multi-agent pipeline on all 1,000 examples with two controls: a two-call form-planning stage that receives no catalogue information (Two-call planning), and the same catalogue-card format with randomly selected entries (Random cards). The following results are assessed on Qwen3.6-27B using the revised judge.

SystemValue transformationChart formOverall
Multi-agent pipeline68.259.372.4
Two-call planning73.662.074.2
Random cards76.067.274.5
+Catalogue retrieval (Ours)81.071.775.5

Additional planning alone yields a modest improvement. With the number of calls matched, catalogue retrieval produces a larger improvement, pronounced more on value transformation and chart-form selection.

Random catalogue cards also improve performance over planning without catalogue information, suggesting that even untargeted catalogue content can provide useful guidance. Relevant retrieval nevertheless outperforms random cards, with the clearest gains again in value transformation and chart-form selection.

Together, these controls indicate that the retrieval gains are not fully explained by an additional planning call or catalogue access alone; selecting content relevant to the request provides further benefits. We will add the criterion-level results for all conditions to the revised paper.

(W2/Q5) Explicit requests and data-location information

The explicit condition uses the fully specified query written during benchmark construction, before it was rewritten into an implicit query. This explicit query specifies the data selection, transformation, and visualization requirements, and can therefore disclose the target table, rows, or columns.

A sample from the benchmark is:

Explicit: "Create a composition chart showing the breakdown of the total Construction Costs ($10,898,000) by responsible entity (Federal, State, County, Town). Calculate and display the percentage share of each entity's contribution to the total construction cost. Exclude the 'Total' row and the 'Annual Operation and Maintenance Costs' section from the chart."

Implicit: "Can you chart the breakdown of total construction costs by responsible entity so I can see at a glance what share each government body contributes?"

We agree that the original implicit–explicit comparison alone does not isolate analytical and visual specification from data-location information. We therefore extended the experiment reported in Table 5 by varying both query form (implicit vs. explicit) and input context (full document vs. oracle table only).

QueryContextTask fulfillmentData correctnessReadabilityOverall
ImplicitFull document46.376.966.462.3
ImplicitOracle table48.784.466.366.5
ExplicitFull document85.476.964.578.7
ExplicitOracle table86.279.662.580.0

Please note that the result above use the revised judge, which is why the result is different with response of the identical component for reviewer E8Kw.

Providing the oracle table improves data correctness under both query forms but has little effect on task fulfillment. In contrast, making the request explicit substantially improves task fulfillment regardless of whether the model receives the full document or only the oracle table.

Thus, the task-fulfillment gap remains nearly unchanged even when the relevant table is supplied directly. This supports our finding that inferring the analytical and visual requirements is a major bottleneck, while document handling introduces an additional difficulty reflected primarily in data correctness. We will incorporate this experiment and distinction into the paper.

(W3) Definitions and case-study presentation

We used analytical operation alongside value transformation because the latter alone may not fully convey the intended scope. However, the distinction was unclear and may have caused confusion. Value transformation is defined as deriving the quantities needed to answer the request, including retaining appropriate raw values when no numerical transformation is required. We will use value transformation consistently throughout the revised paper and clarify this definition.

We also agree that the case study should be more comprehensible. This was due to the page limit and we will provide larger versions of the chart pair including the exact user request, which was "Can you chart the breakdown of depreciation expenses for 2020 and 2019 so I can see at a glance how the cost structure shifted between the two years?".

(Q3) Information available to human raters and the judge

The human raters were shown the oracle table in addition to the source document, request, and rendered charts. This was omitted from Appendix C.3, and we will correct the description.

The remaining differences are the chart-generation code and sample-specific criterion rubrics. The rubrics are a core contribution of our LLM judge and part of the evaluator being validated against independent human judgments. Code is provided to help the LLM assess chart details through a representation it is familiar with, whereas human raters evaluate the rendered charts intuitively. We therefore omitted the code for the human verifiers.

(Q6) Task selection and chart-form tiers

The ten task types were curated through careful manual review of source PDFs to identify and group recurring analytical scenarios in which users express a goal without specifying the required operations or visual representation. Chart-form tiers were initially generated by an LLM based on how well each form supports the intended analysis, and human experts reviewed them for appropriateness.

Updated evaluation results

The tables below report the revised evaluation for the table from the paper.

Updated Table 2.

Modelvalue transformationanswer visibilitychart formtask fulfillment (avg)data selectionvalue fidelitylabel-encoding fidelitydata correctness (avg)presentationoverall
GPT-5.6-Sol82.092.468.781.086.790.284.086.994.785.5
Claude Opus 587.287.176.383.588.090.876.885.296.486.1
GLM-5.383.584.571.179.786.888.582.385.893.384.3
KIMI K2.671.282.761.271.785.387.867.580.291.178.1
DeepSeek-V4-Flash55.165.245.555.387.289.373.583.383.971.4
Gemma4-31B53.553.447.051.385.792.369.882.670.867.5
Gemma4-26B-A4B53.057.145.051.782.589.571.581.174.767.6
Qwen3.6-27B49.549.540.046.380.685.564.676.966.462.3
Qwen3.5-122B-A10B47.160.640.549.482.486.662.277.167.463.8
Qwen3.5-35B-A3B44.260.837.647.577.081.159.772.665.060.8
Qwen3.5-27B52.255.944.951.083.888.565.279.266.665.3
GPT-OSS-120B59.057.946.854.682.485.857.575.264.064.8
Nemotron 3 Super44.047.339.043.480.085.865.977.264.260.9
Mistral Small 446.848.541.145.476.381.254.070.558.358.0

Updated Table 3 for the Qwen baseline.

SystemTask fulfillmentData correctnessReadabilityOverall
Single-shot (Qwen3.6-27B)46.376.966.462.3
Multi-agent pipeline60.982.078.772.4
Catalogue retrieval (Ours)70.581.572.975.5

Updated Table 4.

task fulfillmentdata correctnessreadabilityoverall
Ours.383.239.401.601
w/o decomposition.148.085.401.477
w/o rubrics.241.219.401.492
w/o criterion-wise.341.152.323.442
Human vs. human.282.048.367.480
Reviewer_E8KwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ID2C evaluates chart generation from real documents when users specify analytical goals without explicitly naming the required transformations or chart forms. The benchmark contains 1,000 requests across ten analytical task types. It includes diagnostic, sample-grounded evaluation rubrics and a retrieval-augmented multi-agent harness supported by a catalogue of chart forms and analytical measures.

Critical strengths
  1. Translating implicit analytical goals into transformations and chart forms is a meaningful capability. Real-document inputs and diagnostic rubrics make ID2C a potentially useful benchmark.

  2. The implicit-versus-explicit comparisons expose planning difficulty, and the catalogue-supported harness offers a practical response. Human-alignment evidence provides useful initial support for the evaluator.

Critical weaknesses
  1. Acceptable alternatives. An implicit analytical request can admit multiple defensible transformations or chart forms. Sample-specific anchors should not penalize coherent alternatives simply for differing from the construction-time choice. Their treatment is important to the benchmark's validity.

  2. Reference verification. The abstraction-based dimensions and human screening are useful. However, judge-human alignment does not by itself establish that intended transformations and answers are correct. Independent solutions or an ambiguity audit would provide more direct evidence.

  3. Diagnostic scope. The implicit-versus-explicit comparison establishes planning difficulty. Separating document-context errors from general transformation and chart-planning errors would clarify what the benchmark diagnoses. This is a question about the resource's empirical value, rather than requiring a new generation algorithm.

Questions for authors
  1. How do rubrics recognize multiple defensible transformations and chart forms?
  2. What independent checks support the intended answers and their ambiguity handling?
  3. Which findings distinguish document-context failures from general analytical-planning failures?
1 comment

We thank the reviewer for recognizing the importance of the problem and the value of our benchmark and catalogue, and for the thoughtful feedback on evaluation, experimental controls, and presentation. We address these concerns below.

(W1/Q1) Accepting alternative transformations and chart forms

We agree that an implicit analytical request may admit multiple valid solutions, and that recognizing such alternatives is important for a valid evaluation. We would like to clarify that our rubrics are already designed to accommodate alternatives for various chart forms and transformations.

For chart-form selection, rubrics score multiple alternative forms according to how well they address the user’s request (example rubric: “Score 3: A pie chart, donut chart, or single 100% stacked bar chart; Score 2: a treemap, waffle chart, or free-standing bars of the percentages…”). For value transformation, the prompt accepts mathematically equivalent formulations, and sample-specific rubrics can allow distinct transformations (example rubric: “a derived dispersion metric … such as standard deviation (sample or population convention accepted as equivalent) or coefficient of variation”).

Although these rubrics may not exhaustively cover every valid alternative, they explicitly recognize multiple transformations and chart forms, reducing the risk of penalizing a valid solution.

(W2/Q2) Reference Verification

We agree that judge-human alignment does not independently establish the correctness of the benchmark references. This is addressed separately during benchmark construction. As described in Appendix A.1, expert reviewers have inspected each candidate request together with its target table, surrounding context, fully specified internal task, and sample-specific rubric. Candidates with factual inconsistencies, ambiguous scope, or incomplete rubrics are rejected. This review process helps reduce reference errors and ambiguity in the rubrics used by the LLM judge. We welcome suggestions for additional checks and would be happy to conduct further validation to strengthen this evidence.

(W3/Q3) Distinguishing document-context and analytical-planning difficulties

We agree that separating these factors would clarify what the benchmark diagnoses. We therefore extended the experiment reported in Table 5 by varying both query form (implicit vs. explicit) and input context (full document vs. oracle table only).

QueryContextTask fulfillmentData correctnessReadabilityOverall
ImplicitFull document46.375.460.060.7
ImplicitOracle table48.782.659.364.7
ExplicitFull document85.473.160.276.5
ExplicitOracle table86.275.957.577.7

Providing the oracle table improves data correctness (3~7%p) under both query forms but has little effect on task fulfillment. In contrast, making the request explicit substantially improves task fulfillment (37~39%p) regardless of whether the model receives the full document or only the oracle table.

Thus, the task-fulfillment gap remains nearly unchanged even when the relevant table is supplied directly. This supports our finding that inferring the analytical and visual requirements is a major bottleneck, while document handling introduces an additional difficulty reflected primarily in data correctness. We will incorporate this experiment and distinction into the paper.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers found the problem we address important (E8Kw, gfKE, gF74), ID2C a useful and carefully structured benchmark (E8Kw, gfKE, gF74), and our catalogue a useful resource (E8Kw, gfKE). They also noted that our analysis exposes planning as a key difficulty (E8Kw) and that our human study supports the evaluator (E8Kw, gF74).

During the response period, we added the following analyses and experiments.

  1. We vary both the request form and the input context. Giving only the relevant table mainly improves data correctness, while explicit requests substantially raise task fulfillment in either context, confirming that inferring implicit user requirements is the main bottleneck (E8Kw, gfKE).
  2. We revise the rubrics so that valid transformations and clean default styling are no longer penalized. Rescoring all outputs leaves the system ranking almost unchanged (gfKE).
  3. We report the full human study with confidence intervals and find that the judge agrees with human raters about as well as the raters agree with each other (gfKE).
  4. We compare catalogue retrieval against a planning baseline with matched calls and against random catalogue entries. Our harness performs best, with gains concentrated on value transformation and chart form (gfKE).
  5. We contrast example requests from prior benchmarks with ID2C, showing that only ID2C leaves both the transformation and the chart form unstated over diverse real documents (gF74).
  6. We compare our harness with Doc2Chart and MatPlotAgent under the same backbone, and ours achieves the best overall score, with the clearest margin in task fulfillment (gF74).

These results address the reviewers' concerns, support ID2C as a benchmark for inferring implicit user requirements, and show that our catalogue-based harness improves performance on this task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.