- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether chart question answering can be improved by optimizing the inference program around a frozen VLM, without retraining. Gemini 2.5 Flash uses failure examples to evolve prompts, question-type-specific hints, generation budgets, and output post-processing. Candidates undergo static validation, with parents selected by training accuracy and the final program selected by holdout accuracy. Experiments cover three VLMs and four chart benchmarks, reporting an increase in average descriptive exact-match accuracy from 16.6% to 41.3% across twelve combinations. A ChartQA case study also compares against a hand-tuned program and reports a 3.8× throughput improvement over the seed. The paper attributes the benefits partly to shorter, better-formatted answers.
Critical strengths
-
Optimizing prompts, generation budgets, and answer extraction together is a practical approach that requires no model retraining. Searching over executable code allows these components to be adjusted jointly rather than tuning the prompt alone. The resulting inference program can be reused without running evolutionary search for each new question, making it potentially useful for improving existing VLM applications with limited training resources.
-
Experiments cover twelve model–benchmark combinations across three VLMs and four datasets, all showing improvements over their respective seeds. This provides broader empirical coverage than a single-model or single-benchmark demonstration. The efficiency analysis also shows that higher exact-match accuracy can accompany shorter, faster generation, highlighting the practical importance of inference-pipeline configuration for both accuracy and deployment efficiency.
Critical weaknesses
-
The additional value of evolutionary search is insufficiently established. The seed is explicitly described as having accuracy “low by design,” without formatting hints or post-processing. In the only stronger comparison, the hand-tuned program achieves 56.3% accuracy and 1.20 queries/s, versus 58.6% and 1.24 queries/s for best_speed. These differences are considerably smaller than the headline gains. There are no prompt-optimization comparisons or component ablations separating prompt changes, token budgets, and post-processing. A larger search space alone does not establish an advantage over simpler optimization methods. Moreover, Table 3 includes incorrect numerical predictions, so these examples cannot establish that formatting corrections account for most of the improvement.
-
The evaluation does not clearly establish independent generalization. The holdout window is repeatedly used for program selection and therefore serves as a validation set. Table 1 evaluates the “full descriptive portion” without clearly stating whether search and selection examples are excluded; Table 4 uses 128 descriptive questions, matching the combined training and holdout sizes. The 26-question holdout is also small, and no variability across independent searches is reported. The relationship between the two evaluations also needs clarification: the seed’s descriptive EM is 0.5% on the full ChartQA descriptive portion in Table 1, versus 38.3% on the 128-question window in Table 4. Although the evaluation sets differ, the paper does not explain the window selection or whether other evaluation settings change. Finally, results under the official benchmark protocols would help establish whether the gains extend beyond answer-format compliance.
Questions for authors
-
What are the search-training, selection-validation, and final-test splits and sample counts for each benchmark? Table 1 is computed on the "full descriptive portion" (described as "not only the holdout window"), while the 128 descriptive and 64 reasoning questions in Table 4 match the combined training and holdout sizes (102 + 26 and 51 + 13). Do these evaluations include examples used for search or program selection? Could you report results on a test set fully disjoint from both?
-
The Qwen3-VL-2B/ChartQA seed scores 0.5% in Table 1 but 38.3% in Table 4. Do the seed code, prompts, answer normalization, or question sampling differ between these evaluations? If only the evaluated subset changes, how was the Table 4 subset selected?
-
How much improvement remains over a simple baseline with a short-answer instruction, a fixed answer extractor, and a comparable generation budget? A comparison with a prompt-only optimizer using comparable numbers of optimizer calls and candidate evaluations would help establish the additional value of program-level search. Controlled ablations of prompt changes, token budgets, and post-processing would also clarify which components drive the gains.
-
The qid examples in Sec. 3.1 ("highest y-tick", "number of series") resemble CharXiv's templated descriptive questions. How are the descriptive subsets and qid labels obtained for ChartQA, PlotQA, and EvoChartQA, and are they available at deployment time? Do any evolved per-qid rules encode benchmark-specific answer patterns? Separately, how is the answer normalized, and do the gains hold under each benchmark's official evaluation protocol?
-
Table 3 shows incorrect values in the seed outputs, such as 58 instead of 60 and 28 instead of 38, while the evolved programs return the correct answers. These displayed differences are not explained by removing extra text alone. Since the seed outputs are abbreviated and the evolved answers may include post-processing, a worked example with the relevant code changes and full raw VLM outputs would help clarify whether the correction comes from a changed model response, answer extraction, or another post-processing step.
We thank the reviewer for the valuable review. The requested held-out, official-metric comparisons are included, with all changes marked in blue.
Comment: What are the search, selection, and final-test splits? Are the Table 4 samples reused?
Response: In our observed and submittes study, yes. Search uses 102 descriptive questions, selection uses 26, and Table 4 scores their union of 128 (51 + 13 = 64 for reasoning). The holdout chooses the saved program, so it is a validation set, and we now write this and clarify explicitly. Table 1 is the full descriptive portion of each benchmark (seed / evolved):
We also do a test on a 1,000-question subset of the official ChartQA test set (500 human-written and 500 automatically generated questions, one per distinct chart).
Comment: Why is the ChartQA seed 0.5% in Table 1 but 38.3% in Table 4?
Response: Thank you for catching this. The two cells are the same seed scored on two different sets, the full ChartQA descriptive portion (0.5%, Table 1) and the 128-question case-study window (38.3%, Table 4). The gap is not an improvement, and the paper now labels each cell by its set. The performance claim now comes applied on the shared held-out test, scored with relaxed accuracy, the recorded metric is desccribed as follows:
Comment: What remains beyond short-answer prompting, extraction, and token limits? Compare with prompt-only optimization at a matched budget.
Response: We now ran this comparison at the same candidate-evaluation budget, where the prompt-only arm may change only the prompt. The evolved program is 5.5 points above prompt-only optimization on the 2B model and 2.5 points above it on the 32B model (95% bootstrap CI [+0.9, +4.1]). The manual wrapper already includes a short-answer instruction, an extractor and a token limit, and the evolved program is 7.5 and 3.5 points above it (CI [+1.7, +5.3]). Restoring the manual prompt, extractor or token budget costs 4.2, 2.7 and 1.1 points.
Comment: Explain descriptive subsets, qid labels, normalization, and the official evaluation protocols.
Response: The descriptive subsets are the descriptive questions of each benchmark. qid is the question-type identifier as observed in Section 3.1. CharXiv supplies it with its questions, and on the other three benchmarks it is the type the question itself asks for, so we get it at deployment. The gold answer is never an input. Normalization is each benchmark's own. We now begin with the official metric, keep strict exact match as a diagnostic, and corrected our statement on CharXiv, whose released evaluator uses an LLM judge.
Comment: Table 3 changes incorrect numerical values. How can these examples support a formatting explanation?
Response: They cannot, and you are in-fact right to reject that reading.Thank you so much. The rows of Table 3 are:
Removing the words around 58 does not produce 60, and removing the percent sign from 28% does not produce 38, so the returned value itself changed. We rewrote this paragraph to make it more clear. The experimental run separates the two mechanisms. Restoring the manual prompt costs 4.2 points, so the generation changes with the prompt, and restoring the manual extractor costs 2.7 points, so extraction contributes after the generation.
Comment: The evidence for accuracy, speed, and generalization is weaker than the headline suggests.
Response: We agree, and now write each claim with a comparison supporting it. On accuracy, the claim is the held-out table above, where the evolved program beats both a competent manual wrapper and matched prompt-only optimization on two model sizes. The first automated edit only matches prompt-only optimization, so the gain is not a single short-answer edit. On speed, best_speed is 3.8$\times$ faster than the seed and about 3.3% faster than the hand-tuned program. On reasoning exact match, which was never the fitness, the hand-tuned program leads (17.2% against 14.1%), and we report that as a loss.
We sincerely thank the reviewer for their efforts and constructive feedback, and are happy to clarify anything further.