acceptodds
Under review as a conference paper at ICLR 2027

AgentCypher: Learning to Revise and Accept Text-to-Cypher Queries from Execution Evidence

Abstract

Text-to-Cypher translates natural-language questions into executable queries for retrieving information from property graphs. A key challenge is that queries can execute successfully and return nonempty results despite semantic errors, such as following relationships in the wrong direction.A ReAct formulation makes query execution part of the reasoning process, allowing the model to inspect results and revise its entity choices and graph patterns. We propose AgentCypher, a reinforcement learning framework that integrates query execution, verification, and revision to improve the accuracy of generated Cypher. Specifically, the Grounded Query Execution (GQE) module tests graph patterns constructed from selected schema elements and grounded entities against the target database. The Execution-Guided Verification (EGV) module examines these results and query-specific graph evidence for potential errors in entity grounding, relationship direction, and result multiplicity, guiding query revision and acceptance. The Graded Terminal Reward (GTR) module compares the accepted query’s results with reference results, distinguishing exact answers from partial results through penalties for missing, extraneous, and duplicate rows. AgentCypher achieves 82.41% execution accuracy and 88.83% provenance subgraph Jaccard similarity on CypherBench, and 42.13% execution accuracy on Neo4j-Text2Cypher.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Re-act, Cypher Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_niCCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Text-to-Cypher, where a model turns a natural-language question into a query over a property graph. It points out that a query can run and return rows while still being wrong, for example by following a relationship in the wrong direction. The proposed AgentCypher trains a 7B model with reinforcement learning to work in several turns. It looks up entity names, runs candidate queries, receives extra check results (whether the reversed pattern also matches, where an empty result fails, and whether rows repeat), and then revises or accepts the query. During training, a graded reward gives partial credit based on missing, extra, and duplicate rows. AgentCypher reaches 82.41% execution accuracy on CypherBench, compared with 69.59% for Cypher-RI, and 42.13% on Neo4j-Text2Cypher.

Critical strengths
  • The method gives strong results for a 7B model, clearly above the other Text-to-Cypher methods compared, on two benchmarks.
  • The paper includes a detailed analysis of the model’s behavior, showing when it revises, which checks lead to fixes, and which errors remain. This makes the method’s limits clear.
Critical weaknesses
  • The novelty is limited, and the one new part adds little. Prior work has already used execution feedback to revise queries over several turns, trained Text-to-Cypher with execution-based rewards, and handled relationship direction by building queries from relations that exist in the graph. The new part is the set of check queries in EGV, but they raise execution accuracy only from 78.40% to 80.81% (Table 2). They also cover a narrow set of errors: they do not check whether the query includes every condition in the question. Accuracy on time-sensitive queries is only 41.67%, compared with 92% for Cypher-RI, and 185 of the 297 accepted incorrect queries receive no report from any check (Appendix C).
  • The evaluation ignores the extra computation at inference time. AgentCypher allows up to eight query executions per question plus extra checks, but the paper does not report database calls, latency, or cost. This weakens the motivation, which is the high cost of large-model agents, since the paper never compares with them. Several baseline scores are also taken from earlier papers with different setups, so it is unclear how much of the gain comes from having more chances to revise. All results come from single runs.
Questions for authors
  • How does AgentCypher compare with the large-model agent methods that motivate the work, in both accuracy and inference cost (number of turns, queries, and time per question)?
  • Is the model trained on Neo4j-Text2Cypher, or only on the CypherBench graphs?
  • Why is GRPO used? How does it compare with supervised fine-tuning on successful trajectories, or with the same agent loop and checks used by a larger model without training?
Reviewer_mAy7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a practical challenge in text-to-Cypher generation: a query can execute successfully and return results while still answering the wrong question. AgentCypher trains a 7B model to look up entities, execute candidate queries, and decide whether to revise or accept them. Its main contribution is to supplement execution feedback with graph-specific checks on relationship direction, path matching, entity use, and result multiplicity, alongside a terminal reward that distinguishes partially correct answers. I find the focus on inspecting how an answer was obtained particularly useful, since execution success alone provides little assurance of semantic correctness. Experiments on CypherBench, including evaluation on unseen graphs, and Neo4j-Text2Cypher show improved answer accuracy over the reported baselines. Ablations and trajectory analyses provide further evidence that the model can use this feedback to correct query errors.

Critical strengths

The paper addresses a relevant limitation of text-to-Cypher generation: a query may execute successfully while answering the wrong question. Using graph-specific checks to guide revision is a reasonable approach to this problem. The reported results improve over the compared baselines on both benchmarks, including unseen graphs in CypherBench, providing evidence that the overall approach is useful.

Critical weaknesses
  1. The contribution of the learned policy is not sufficiently isolated from that of the tools. AgentCypher combines entity lookup, execution feedback, hand-designed graph checks, and reinforcement learning. The experiments show that the complete system works, but leave unclear how much the proposed training contributes beyond the information supplied by the tools. For example, EGV provides concrete clues about relationship direction and duplicate nodes that may make the required edits relatively straightforward. A comparison with the same backbone without the proposed RL training, using identical tools, prompts, and inference budgets, would help establish the value of learning to revise and accept queries.

  2. Some component-level gains are small, and their stability is unclear. In Table 2, the difference between execution success rates of 99.64% and 99.87% amounts to roughly five or six questions out of 2,348. In Table 4, GTR improves EX from 81.54% to 82.41% over the result-level partial reward baseline (Papicchio et al., 2026), a gain of 0.87 percentage points. Without results across independent training seeds or uncertainty estimates, it is difficult to tell whether these differences are reliable.

  3. The resource and efficiency benefits of using a smaller model remain unverified. Lower deployment and inference costs motivate the work, but AgentCypher requires multiple generation turns, repeated processing of interaction history, and additional database queries. These costs may offset the savings from a smaller backbone. The paper does not report actual token usage, database calls, end-to-end latency, or peak memory, and it is unclear whether all EGV checks count toward the execution budget. A comparison at matched resource budgets, or a cost comparison at similar accuracy, is needed to assess the practical efficiency of the approach.

Questions for authors
  1. How much of the improvement depends on the proposed RL training? Have you evaluated the same backbone without this training, using the same entity lookup and EGV tools, prompts, and inference budgets?

  2. How do you distinguish gains from the proposed method from variation due to training randomness? Have you repeated the experiments with independent training seeds, and can you report the mean and standard deviation to establish whether the improvements are consistent?

  3. How do you assess whether the smaller model retains a resource advantage once the full interaction process is included? For example, comparisons with the baselines in per-question token usage, database calls, and end-to-end latency could help clarify this.

Reviewer_dCcfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AgentCypher, a reinforcement-learning framework for generating Text-to-Cypher queries through iterative execution, verification, and revision. It combines three components: Grounded Query Execution (GQE), which performs entity lookup and query execution; Execution-Guided Verification (EGV), which provides additional graph evidence concerning relationship direction, entity grounding, and result multiplicity; and Graded Terminal Reward (GTR), which distinguishes partially correct query results during reinforcement learning. On CypherBench, AgentCypher achieves 82.41% execution accuracy and 88.83% provenance subgraph Jaccard similarity, outperforming the compared methods. It also obtains 42.13% execution accuracy on Neo4j-Text2Cypher. The authors provide component ablations, comparisons with alternative agent interaction settings and reward designs, and performance analyses across different graph-query patterns.

Critical strengths

S1. Text-2-Cypher is an important problem for high-quality graph retrieval. The paper makes reasonable observation that executable queries may not necessarily be semantically sound or complete. Execution-guided revision is a practical response.

S2. The experimental study has provided empirical improvements with ablation study.

Critical weaknesses

W1. The overall technical novelty seems limited. The novelty relative to prior execution-feedback and RL approaches needs stronger justification. The framework builds on ReAct-style query revision, entity grounding, execution verification and reward shaping. Closest work includes Cypher-RI (Su et al., 2025), Adaptive Text2GQL (Niu et al., 2026) and LAST-CQ (Prokopiou et al., 2026), as discussed in Sections 1–2. The paper does include related baseline comparisons.

My concern is more on identifying the new mechanism beyond combining familiar components. As is, the proposed modules seem quite straightforward extension of existing techniques. For example, EGV performs predefined checks, and the acceptance gate is prompt-specified rather than programmatically enforced (Section 3.3). More precise novelty and cost analysis would strengthen the claim.

W2.The verification coverage and tests on important patterns seems not complete. EGV checks selected matching, direction, multiplicity and entity conditions, but not arbitrary Cypher semantics (Section 4.3). AgentCypher reaches only 41.67% accuracy on time-sensitive queries, compared with 92% for Cypher-RI, and underperforms Cypher-RI on two-relation patterns involving the same node pair. These are meaningful query classes. Category sample sizes, detailed failure cases and targeted checks are needed to assess robustness.

Questions for authors

Q1 (W1, Section 3.3, Eq. 12). How is the verification gate learned, and how does it differ from a prompted ReAct agent? What evidence shows that the policy follows the verification protocol when acceptance is not programmatically blocked?

Q2 (W1, Tables 2–4). Could the authors compare the closest execution-feedback methods under matched backbone, training data, interaction budget and tool-call limits to isolate graph-specific checking from more executions?

Q3 (W2, Section 4.3). How many queries are time-sensitive, why does this class regress so sharply, and could checks of temporal predicates or validity intervals address it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.