- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
GPBench contains 1,000 graph-analysis problems for evaluating workflow decomposition, algorithm selection, and parameter and dependency resolution. The authors provide execution-validated reference workflows and evaluate domain-guided prompting and hierarchical algorithm retrieval, both independently and in a three-stage pipeline.
Critical strengths
The execution-validated reference workflows and separate evaluation of decomposition, algorithm selection, and parameter binding provide a useful resource for diagnosing errors in graph-tool orchestration.
Critical weaknesses
The writing and figures need substantial improvement. Many sentences have a formulaic, AI-like style that makes the paper unnecessarily difficult to read. Several figures, particularly Figure 2, also give the impression of rough AI-generated drafts rather than carefully prepared scientific illustrations. My concern is not AI use itself, but the insufficient editing and visual polish.
The benchmark largely asks models to recover a workflow already described in the question, rather than determine how to solve a high-level analytical problem. Questions are generated from preselected workflows while preserving their operations, parameters, and dependencies, even when algorithm names are omitted. This makes the task closer to semantic parsing and algorithm identification. The benchmark may be useful for that narrower purpose, but its value as an evaluation of autonomous graph-analysis planning is insufficiently established.
Questions for authors
What substantive planning decisions remain open to the model once the question specifies the required operations and dependencies? How does GPBench distinguish planning ability from recovering an already-described workflow?
2 comments
Responses to Reviewer 31np (2/2)
Q1 / W2. What decisions does the model need to make?
In GPBench, planning is required throughout graph analysis, including problem decomposition, algorithm selection, and dependency binding.
Even when a question specifies the analytical requirements, operations, and relationships between stages, the model must still map these requirements to concrete graph operations, select algorithms based on graph properties and downstream needs, and determine how intermediate results are used in subsequent computations. For example, the model must decide whether an upstream node set should be used to construct the input subgraph for a downstream algorithm or bound to its source-node parameter. Our end-to-end experiments further evaluated these decisions and their implementation: as shown in Table 1, with IRA enabled, GPT-5.5 achieved a workflow accuracy of 53.10% and a final-answer accuracy of 45.10%. We also observed that some generated programs mishandled intermediate results passed between stages.
Table 1. Workflow and final-answer accuracy of our three-stage pipeline on the dataset.
| Model | Workflow Accuracy (w/o IRA) | Workflow Accuracy (w/ IRA) | Final-answer Accuracy (w/o IRA) | Final-answer Accuracy (w/ IRA) |
|---|---|---|---|---|
| Qwen3-8B | 6.50% | 7.00% | 2.60% | 2.70% |
| Qwen3.8-Flash (Non-Thinking) | 41.90% | 44.30% | 35.20% | 36.80% |
| Qwen3.8-Flash (Thinking) | 47.90% | 48.10% | 39.40% | 39.50% |
| DeepSeek-V4-Flash | 46.90% | 52.20% | 39.40% | 41.80% |
| GPT-5.5 | 48.80% | 53.10% | 42.40% | 45.10% |
Responses to Reviewer 31np (1/2)
W1. Improving the writing and figures
Thank you for pointing out these issues with the writing and figures. We have edited the experimental discussion to reduce repetitive, overly general, and formulaic phrasing. Based on the additional model comparisons and specific errors observed in the experiments, we have revised the analyses of task decomposition, algorithm selection, and parameter binding, and added an analysis of the end-to-end experimental results. We have also revised the relevant figures, with particular attention to Figure 2, to improve their clarity and visual presentation.