- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies Text-to-Cypher, where a model turns a natural-language question into a query over a property graph. It points out that a query can run and return rows while still being wrong, for example by following a relationship in the wrong direction. The proposed AgentCypher trains a 7B model with reinforcement learning to work in several turns. It looks up entity names, runs candidate queries, receives extra check results (whether the reversed pattern also matches, where an empty result fails, and whether rows repeat), and then revises or accepts the query. During training, a graded reward gives partial credit based on missing, extra, and duplicate rows. AgentCypher reaches 82.41% execution accuracy on CypherBench, compared with 69.59% for Cypher-RI, and 42.13% on Neo4j-Text2Cypher.
Critical strengths
- The method gives strong results for a 7B model, clearly above the other Text-to-Cypher methods compared, on two benchmarks.
- The paper includes a detailed analysis of the model’s behavior, showing when it revises, which checks lead to fixes, and which errors remain. This makes the method’s limits clear.
Critical weaknesses
- The novelty is limited, and the one new part adds little. Prior work has already used execution feedback to revise queries over several turns, trained Text-to-Cypher with execution-based rewards, and handled relationship direction by building queries from relations that exist in the graph. The new part is the set of check queries in EGV, but they raise execution accuracy only from 78.40% to 80.81% (Table 2). They also cover a narrow set of errors: they do not check whether the query includes every condition in the question. Accuracy on time-sensitive queries is only 41.67%, compared with 92% for Cypher-RI, and 185 of the 297 accepted incorrect queries receive no report from any check (Appendix C).
- The evaluation ignores the extra computation at inference time. AgentCypher allows up to eight query executions per question plus extra checks, but the paper does not report database calls, latency, or cost. This weakens the motivation, which is the high cost of large-model agents, since the paper never compares with them. Several baseline scores are also taken from earlier papers with different setups, so it is unclear how much of the gain comes from having more chances to revise. All results come from single runs.
Questions for authors
- How does AgentCypher compare with the large-model agent methods that motivate the work, in both accuracy and inference cost (number of turns, queries, and time per question)?
- Is the model trained on Neo4j-Text2Cypher, or only on the CypherBench graphs?
- Why is GRPO used? How does it compare with supervised fine-tuning on successful trajectories, or with the same agent loop and checks used by a larger model without training?