- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
PPTBench contains 500 scientific diagram reconstruction tasks requiring editable PPTX outputs. The paper introduces a four-stage evaluation and compares 36 model–harness–effort configurations.
Critical strengths
The task is useful, and the evaluation is substantial. The paper also reports human agreement and judge repeatability.
Critical weaknesses
- Evaluator reliability does not adequately support the scoring scheme.
Human agreement is reported on the same cases used for judge selection, without an independent validation set. In Table 2, the judge disagrees with the human majority on 27 of 200 cases, including 15 human-approved outputs that are incorrectly assigned zero. Hard gating turns a local judgment error into complete failure, while the existing stability analyses do not adequately establish how these errors affect different models.
- The large evaluation leaves important questions about execution, cost comparability, and model coverage.
The paper identifies Codex, Claude Code, and OpenCode but does not sufficiently document service access and execution settings for every configuration. Cost accounting combines recorded charges and estimates and needs a consistent breakdown. Moreover, 33 of the 36 configurations come from OpenAI and Anthropic, while Kimi, Qwen, and GLM each receive only one. This limits the scope of conclusions about model capabilities and reasoning effort.
Questions for authors
- Figure 1 uses AI-generated artwork without clearly distinguishing illustrative content from actual benchmark samples. Decoration takes up substantial space, while the examples are difficult to inspect. Please replace it with readable, traceable task examples.
- Do the harnesses connect to official APIs, third-party compatible endpoints, or application sessions? Please report model identifiers, service providers, and software versions for every configuration.
- How are input, output, cached, and reasoning tokens counted? Are retries, failed calls, and tool calls included? Please provide consistent cost accounting and report judging costs separately.
- Why are reasoning-effort sweeps concentrated on OpenAI and Anthropic, with only one configuration for each remaining model? Do the conclusions hold under comparable budgets?
- Can the judge be validated on unseen cases, and can rankings be reported after human correction of gating errors?
Personally, I find it disappointing to see what looks like a direct output from an AI image generator used so prominently in the paper. I am comfortable with AI-assisted figure creation, including using AI to write structured plotting code. The current figure, however, does not reflect the care I expect in presenting a research contribution. Much of the writing also reads like lightly edited AI-generated prose, and several passages are difficult to follow.
The experiments are extensive, but the comparisons span different harnesses and are heavily concentrated on a small number of model providers. I understand that coding subscriptions may make some models more accessible to evaluate. For a benchmark paper, I would nevertheless expect broader model coverage, including models from other providers and competitive open-source models, with comparisons under consistent evaluation settings. I hope the authors do not find these comments discouraging. I would be willing to raise my score if they can adequately address these concerns.
4 comments
Thank you for clarifying the costs. My concern is the allocation of the budget: the reported $3,228.69 spent on the Astra sweep could support several smaller evaluations of models from other providers.
The similar-cost comparisons are useful, but they do not show whether the findings about reasoning effort hold across model families. Testing other families only at high or maximum effort leaves this question open.
I therefore consider this concern unresolved. A small matched-subset effort comparison across additional families would address it more directly, or the conclusions should be limited to the families tested in depth.
Thank you for the constructive suggestion. Following your recommendation, we conducted an effort-level comparison of GLM-5.3-Flash across Low, High, and Max reasoning settings on a randomly sampled 100-task matched subset:
| Metric | Low | High | Max |
|---|---|---|---|
| Overall Score (/100) | 10.54 | 39.81 | 34.86 |
| Valid Artifact Rate | 95/100 | 95/100 | 94/100 |
| Semantic Gate Pass Rate | 18/95 (18.95%) | 48/95 (50.53%) | 40/94 (42.55%) |
| Render Gate Pass Rate | 14/18 (77.78%) | 47/48 (97.92%) | 40/40 (100.0%) |
| All-Gate Pass Rate | 14/100 | 47/100 | 40/100 |
| Mean Detail Score (Passed) | 75.26 | 84.70 | 87.15 |
The results confirm that GLM-5.3-Flash exhibits the same non-monotonic degradation at the highest reasoning effort, peaking at High (39.81) and dropping noticeably at Max (34.86). Closer case inspection reveals a consistent underlying failure mode: while Max achieves a higher detail score on passed samples (84.70 --> 87.15), its semantic gate pass rate drops from 50.53% to 42.55% because High zooms into local regions (such as computational graphs) for direct visual inspection after initial script probing, whereas Max over-relies on scripts for pixel grayscale thresholding, ASCII pixel dumps, and repeated local font/bounding-box measurements while skipping direct visual verification of global topology and edges, resulting in missed or spurious connections. Alongside adding these results to the revision, we will explicitly scope our conclusions on reasoning-effort scaling in the main text and conclusion to the model families systematically evaluated across effort levels.
Thank you again for your thorough review. We hope these additional experiments and revisions address your concern, and we would be grateful if you would reconsider your score.
I think my request was misunderstood. I was asking for a broader comparison across providers and price points, for example by adding more Qwen models and including DeepSeek and Grok. The new GLM reasoning-effort sweep tells us how one model responds to different effort settings. It adds little evidence on how models from different providers compare at similar costs.
I still think the task is useful, but I find the benchmark construction fairly straightforward and see limited novelty in the design. My original assessment and rating stand.
Response to Reviewer ZLN9
We thank the reviewer for the detailed and candid review. We are glad that the reviewer finds the task useful and the evaluation substantial, and values the reported human agreement and judge repeatability. We address each concern below.
W1 & Q5: Judge reliability and the 27 disagreements
The 200 validation samples were indeed also used for judge selection. The resulting bias is limited, and judge errors do not drive the ranking:
W2 & Q2: Service access and execution environment
All configurations use APIs; no app sessions were used. GPT and Claude models go through a pass-through gateway to the official OpenAI and Anthropic endpoints, which preserves model identifiers, native reasoning parameters and token logs; Qwen, Kimi and GLM use their vendors' official APIs. Model identifiers are gpt-6-astra, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, claude-opus-5, claude-sonnet-5, qwen3.8-max, kimi/kimi-k3 and glmcustom/glm-5.3-flash; the judge is gpt-5.6-luna (high) via Codex. Appendix J now lists these details for every model and effort setting.
W2 & Q3: Token accounting and judging cost
All costs are now computed with one formula: logged token usage times the official list price on the experiment date, so configurations are directly comparable.
In total, generation costs $11,146.13 and judging $1,917.71. Per-configuration generation costs are reported in Table 1, and the accounting rules and prices in Appendix J.
W2 & Q4: Model coverage and similar budgets
Q1 and writing
We accept this criticism. The AI-generated illustration has been removed: Figure 1 now shows ten real source figures, one per domain, and Figure 4 has been redrawn with readable labels. We have also revised the prose to state procedures and results directly.
We thank the reviewer again for the valuable comments, which will make the paper more rigorous and readable, and we would be grateful if the reviewer would reconsider the score.