acceptodds
Under review as a conference paper at ICLR 2027

Continual Experience Learning from Expert Feedback for NL-to-Cypher Data Synthesis

Abstract

Natural-language-to-structured-query systems rely on paired natural-language requests and executable queries for training, prompting, and evaluation. Constructing such data manually is expensive, while LLM-based synthesis can introduce semantic errors that are not detected by syntax or execution alone. Moreover, corrections to individual examples are often discarded rather than reused to improve future synthesis. We study this problem primarily in the setting of natural-language-to-Cypher (NL-to-Cypher), where existing benchmarks remain relatively limited in semantic quality and structural coverage. We present CIRA, a continual experience-learning framework that converts expert corrections into reusable, scoped guidance for subsequent data synthesis. Instead of updating model parameters, CIRA compiles feedback into experiences that capture the failure, corrective guidance, and applicability conditions, then consolidates and selectively retrieves them during future generation. On an unseen CypherBench graph, CIRA generates NL-to-Cypher data with an estimated 95.0% validity based on a manually verified sample. Compared with 1,000 template-generated benchmark examples, CIRA produces 680 versus 503 distinct query structures and reduces structural duplication from 50% to 32%. In downstream fine-tuning, combining CIRA-generated and benchmark data yields the highest overall execution accuracy for both evaluated models. Across five rounds of expert feedback, CIRA increases the valid-pair rate from 0.29 to 0.92, while the same generation pipeline without experience memory ends at 0.21. On identical specifications, retrieving raw feedback reaches 0.67, showing that compiling and consolidating feedback provides additional benefit beyond retrieval alone. We further show that the learned experience bank transfers across a new schema, a different generator, and a new domain, improving the valid-pair rate by 0.21–0.55 over the corresponding memory-free pipeline. Finally, transferring eligible experiences from Cypher to SQL improves validity from 0.58 to 0.79, providing evidence that the learned guidance can extend beyond the query language on which it was acquired. The code is available at https://anonymous.4open.science/r/CIRA-3814/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual experience learning, learning from feedback, non-parametric adaptation, synthetic data generation, text-to-Cypher, context optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V9GrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper proposes CIRA, a continual experience-learning framework for synthesizing natural-language-to-Cypher (NL-to-Cypher) training data. CIRA converts human expert corrections into structured, reusable experiences, which are consolidated into an evolving memory bank and selectively retrieved to guide subsequent data generation without updating model parameters. The authors evaluate the framework through iterative feedback learning, cross-schema and cross-domain transfer, and downstream fine-tuning experiments. The results show substantial improvements in expert-judged data validity over static generation and raw-feedback retrieval baselines. The authors further claim that CIRA-generated data improves structural diversity and complements existing benchmark data for downstream Text-to-Cypher training.

Critical strengths

The paper is well motivated by the limitations of existing NL-to-Cypher datasets, particularly their semantic errors and insufficient coverage of complex graph queries. The observation that valuable expert corrections are often discarded rather than reused is also compelling. The proposed CIRA framework provides a reasonable approach to addressing these issues through continual experience learning. The experiments show substantial improvements in data validity, while the cross-schema and cross-domain results suggest promising transferability.

Critical weaknesses
  1. Neither the main paper nor the supplementary material provides a complete case study showing how expert feedback is transformed into reusable experiences and applied to new tasks. Without examining actual experience entries and their reuse, it remains unclear whether CIRA learns generalizable principles or retains task-specific information, potentially including answer-related hints. This is a significant limitation given that experience learning is the paper's central contribution.

  2. The downstream experiments rely on relatively outdated models (Llama-3.1-8B and Qwen2.5-Coder-1.5B), without evaluating modern coding models such as Qwen3 or stronger frontier models. The GPT version used in the transfer experiment is also unspecified. More importantly, CIRA-generated data fails to improve execution accuracy on the independent CB-politics benchmark, even when combined with the original training data. The reported overall gains depend on including CIRA-generated test examples, weakening the evidence for genuine generalization.

Questions for authors
  1. Experience learning: Could the authors provide a complete case study showing the original expert feedback, compiled experience, and its successful reuse? How do they rule out the memorization of task-specific or answer-related information?

  2. Downstream generalization: Why does CIRA-generated data fail to improve performance on the independent CB-politics benchmark? Can the authors provide stronger evidence of generalization under matched training budgets?

  3. Model selection: Why were more recent and stronger models, such as Qwen3-based 7B/8B coding models or frontier models, not evaluated? What exact GPT model/version was used in the transfer experiment?

3 comments

We thank the reviewer. We revised the paper to make the evidence explicit.

W1/Q1. What is actually stored and reused? We add an end-to-end case study showing (i) the original specification and generated query, (ii) expert correction, (iii) compiled experience and applicability scope, and (iv) successful reuse on a later task. CIRA does not store the corrected query as a reusable answer. It stores a generalized rule with conditions governing when the rule applies. The reuse task differs in schema entities and query instance; its success requires applying the learned principle rather than reproducing the source answer.

StepWhereContent
(i) Original specification and queryESGF round 1, Pathfinding“Which non-country regions are affected by natural hazards of type Wildfire?” Query: fixed two-hop match, hazard type > hazard > region.
(ii) Expert correctionVerdict: invalid“Pathfinding means shortest paths, variable length or reachability, and this is a fixed two-hop match, i.e. Multi-step under the wrong label.”
(iii) Compiled experience and scopeManager: add (“no existing rule covers this requirement for the Pathfinding class”)Scope: Cypher-specific; no schema anchor; target: query writer. Trigger: “specifications where primary_class is 'Pathfinding'”. Guideline: “use variable-length relationships, shortestPath(), allShortestPaths() or reachability checks, not fixed multi-hop MATCH patterns.” No hazard, region, label or value from the source pair is stored.
(iv) Reuse on different entitiesESGF round 3, Pathfinding, valid“What are the titles of papers that cite, directly or through an intermediate citation, papers that conduct experiments on the ERA5 observational dataset?” Query uses a variable-length citation path. Papers and citations share nothing with hazards and regions; only the principle carries over.

Effect. Pathfinding valid, rounds 0–4: 0/3, 0/3, 1/2, 3/3, 2/3. Retrieved by 17 later ESGF pairs (11 valid); fired on 9 GCMD pairs (8 valid) and 14 Hetionet pairs (13 valid); never offered on SQL (eligibility filter). This shows cross-schema reuse and scope control.

Consolidation. Experiences are updated, merged, or rejected, not stored verbatim. Of 57 manager decisions: 32 add, 11 update, 1 merge, 13 reject. Reject example: “an exact duplicate of existing rule 'Select constraints for non-trivial aggregation'”. Update-then-merge example: ESGF-anchored “Avoid isolated City geocoder lookups” widened once, then merged into “Connect peripheral entities to core domain concepts” and retired. This retains reusable but scoped guidance.

W2/Q2–Q3. Downstream generalization and model selection. The reviewer is right that the original merged “overall” score did not isolate generalization. We redid the study: (i) training sets are 1,000 pairs, including all 500+500 mixtures (CYQUARK alone has 986); (ii) CypherBench (CB) and CIRA are evaluated on separate 200-example test slices, never merged; (iii) we added three newer architectures (Qwen3.5-9B, Qwen3.5-2B, Gemma-4-E4B) alongside Llama-3.1-8B and Qwen2.5-Coder-1.5B, plus two public synthesis pipelines, CYQUARK and Mind the Query (MtQ), run on the same graph.

Execution accuracy on unseen politics (CB slice / CIRA slice):

Train setQwen3.5-9BQwen3.5-2BGemma-4-E4BLlama-3.1-8BQwen2.5-C-1.5B
CypherBench (CB)62.0/30.049.5/23.046.5/34.047.5/28.530.0/24.0
CIRA41.0/56.538.0/35.033.0/46.034.5/42.018.0/33.0
CYQUARK (986)28.5/17.524.0/13.525.0/24.027.5/16.017.5/14.0
MtQ28.5/30.010.0/19.528.0/30.020.0/24.514.0/15.5
CIRA 500 + CB 50047.5/43.537.5/29.037.0/40.540.0/40.026.5/28.5
CYQUARK 500 + CB 50048.0/21.535.0/17.545.0/23.037.5/19.030.0/17.5
MtQ 500 + CB 50048.0/34.021.0/24.544.5/33.041.0/33.516.5/20.5

The equal-size mixtures address the training-budget concern: gains are not simply from adding more data. On the independent CB-politics slice, CIRA training does not outperform CB training. However, CB is instantiated from only 12 graph-independent templates, and 91% of politics test questions share their exact match-pattern template with the soccer training pool. This overlap creates a strong in-distribution advantage for CB-trained data and limits the slice as a test of broader generalization.

CIRA is instead the strongest single source on the human-style slice across all five models and outperforms CYQUARK and MtQ on both slices. We therefore narrow our claim: CIRA provides complementary, human-style coverage, not a replacement for CB on its own templated distribution. The newer architectures address model selection; we will also specify the exact GPT model/version in the transfer experiment.

Reviewer_V9GrOpenReview ↗

Follow-up on Rebuttal and Score Update

Thank you for the honest response. I appreciate that you directly addressed my concerns, re-ran the full matrix of training setups, and appropriately narrowed your claims. The new end-to-end case study also clarifies how experiences are abstracted and reused.

Based on these substantial updates, I have raised my rating from 1 to 2. I would consider raising my score further if you can address two remaining points in the revised manuscript:

  1. Please integrate all the new results into the revised PDF, and polish the figures and overall presentation for better readability.

  2. While I understand the template overlap issue in CypherBench, the CIRA test slice is still generated by your own pipeline. If you could evaluate the fine-tuned models on even a small set of 20 to 50 complex graph queries independently outside the CIRA pipeline, it would provide compelling evidence for downstream generalization.

Thank you for your genuine engagement with our work and for your constructive suggestions. We have addressed both remaining points and uploaded the revised manuscript.

  1. We incorporated all new experimental results and improved the figures and overall presentation.

  2. We added an independent generalization experiment using 197 complex queries from Mind the Query, spanning four graphs unseen during training. These queries were not generated by CIRA, allowing us to assess generalization beyond CIRA-generated data. The setup and results are reported in Section 4.2 and Table 10.

We appreciate your feedback, which has helped strengthen our evaluation and presentation.

Reviewer_qquyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CIRA, a framework that turns expert corrections into reusable guidance for generating natural-language–Cypher pairs. It improves generation by updating an experience memory while keeping the LLM frozen. Experiments show improved validity over feedback rounds, gains over raw-feedback retrieval, and transfer to new settings. The generated data is more structurally diverse than the template-based baseline and complements existing data in downstream fine-tuning.

Critical strengths

The paper addresses a practical problem: expert corrections often fix individual examples without improving future data generation. CIRA turns these corrections into reusable guidance with explicit applicability conditions. This is a useful design choice because it allows general lessons to transfer while keeping schema-specific advice tied to its original context.

Critical weaknesses
  1. The downstream value of the generated data is not yet clear. In Table 2, replacing benchmark data with CIRA data leaves Llama’s overall accuracy nearly unchanged and reduces Qwen’s. Combining the datasets helps, but also increases the training set size. Since the overall score includes CIRA-generated test questions, equal-size training comparisons and independent evaluation are needed to establish a broader benefit.

  2. The comparisons do not establish CIRA’s advantage over existing experience-learning methods. The main baselines either omit memory or retrieve expert comments verbatim. These comparisons show that processing feedback helps, but do not establish the value of CIRA’s particular approach relative to methods that already extract and manage reusable guidance.

  3. The practical cost of the pipeline remains unclear. The paper reports reduced expert time, but not token usage, model calls, generation latency, or cost per valid pair. Given the multiple generation, checking, and repair stages, it is difficult to assess whether the reduction in human effort translates into lower overall production cost.

Questions for authors
  1. Have you compared the combined training data with an equally sized benchmark-only dataset? Do the gains hold on tests not generated by CIRA?

  2. What does CIRA add beyond existing experience-learning methods? Which comparisons or ablations support that distinction?

  3. Could you report token usage, runtime, and total cost to demonstrate the practical feasibility of CIRA?

2 comments

We thank the reviewer. All numbers below will be in the revised PDF.

W1. Downstream value. The merged “overall” score did not isolate downstream generalization, so we redid the study. (i) All training sets, including 500+500 mixtures, use 1,000 pairs. (ii) CypherBench and CIRA test slices each contain 200 samples and are never merged. (iii) We added Qwen3.5-9B, Qwen3.5-2B, Gemma-4-E4B, and two public pipelines, CYQUARK and Mind the Query (MtQ), on the same graph. Execution accuracy on unseen politics (CypherBench / CIRA slice):

Train setQwen3.5-9BQwen3.5-2BGemma-4-E4BLlama-3.1-8BQwen2.5-C-1.5B
CypherBench (CB)62.0/30.049.5/23.046.5/34.047.5/28.530.0/24.0
CIRA41.0/56.538.0/35.033.0/46.034.5/42.018.0/33.0
CYQUARK (986)28.5/17.524.0/13.525.0/24.027.5/16.017.5/14.0
MtQ28.5/30.010.0/19.528.0/30.020.0/24.514.0/15.5
CIRA 500 + CB 50047.5/43.537.5/29.037.0/40.540.0/40.026.5/28.5
CYQUARK 500 + CB 50048.0/21.535.0/17.545.0/23.037.5/19.030.0/17.5
MtQ 500 + CB 50048.0/34.021.0/24.544.5/33.041.0/33.516.5/20.5

Equal-size mixtures rule out larger training sets as the explanation. CIRA does not beat CypherBench on its own slice, which has substantial train–test template overlap: CypherBench uses only 12 graph-independent templates, and 91% of politics test questions share their exact match-pattern template with the soccer training pool. This favors CypherBench in distribution and limits the slice as a test of broader generalization.

CIRA is the strongest single source on the human-style slice across all five models and beats CYQUARK and MtQ on both slices. Our revised claim is complementary human-style coverage, not replacing CypherBench on its templated distribution.

We use MtQ and CYQUARK only as training comparators, not test slices, due to published and observed validity and naturalness issues.

W2. Existing experience-learning methods. We added Dynamic Cheatsheet (cumulative variant) and ACE using authors' code and Gemini 3.7 Flash. Neither released task-specific memory; each built its own from the same expert feedback on 91 CIRA pairs (ESGF rounds 0-3) used by CIRA. Memories were frozen after round 3 and tested unchanged on 24 ESGF round-4 specifications and three transfer targets, without target feedback. Same expert and criteria. Cells show valid queries out of 24 (wrong in parentheses).

SystemESGF round 4GCMDHetionetformula_1 (SQL)
Static (no memory)510 (3)17 (3)14 (3)
Raw-feedback retrieval16---
Dynamic Cheatsheet13 (2)13 (4)16 (6)13 (6)
ACE12 (3)15 (3)17 (5)14 (6)
CIRA22 (0)22 (1)22 (2)19 (3)

In-domain, both beat no memory (12–13 vs. 5), but trail raw-feedback retrieval (16) and CIRA (22). Out of domain, they gain +3 to +5 on GCMD and stay within one pair of no memory on Hetionet and formula_1, while producing the most wrong queries. With identical feedback, CIRA gains +12, +5, and +5 on the three targets. Neither baseline claimed cross-domain transfer; we test whether reusable feedback transfers. Ablations (response to A2r6) show that scope-aware routing, consolidation (update/merge/reject), and utility weighting each matter: removing one lowers ESGF round-4 validity by 4–5/24.

W3. Cost. We reconstructed all model calls in the 5-round ESGF study at Gemini 3.7 Flash list rates.

Per pair, rounds 0-4StaticRaw retrievalCIRA
model calls2.412.453.97
generation seconds11-1511-1530
$ per pair0.0230.0240.043
$ per valid pair0.0870.0550.064

Static is cheapest per pair but costliest per valid pair: 30/115 (26%) valid vs. CIRA's 76/115 (66%). CIRA costs 1.9x more per generated pair for 2.5x the valid yield. Its extra 1.6 calls per pair come from the specification reviser (0.57) and review-to-memory path (1.02: compiler, router, manager).

As the bank grows, CIRA’s cost per valid pair drops from $0.154 (round 0) to $0.040 (round 4), and corrections from 16 to 2 per round. The study uses ~1,145 calls and 9.2M input tokens. In production, a frozen bank with no expert loop produced 1,000 pairs in 2.7h using 10,243 calls. Model cost per valid pair is $0.064; expert review dominates, but feedback time fell from 56sec to 22sec per pair across rounds.

AuthorsOpenReview ↗

Revision has been uploaded

Thank you again for your review. We have updated the main paper and appendices to address your concerns, along with those raised by the other reviewers. The updated PDF has been uploaded, with new additions highlighted in blue for ease of reference. We would welcome your feedback on these changes. If any of your original concerns remain unresolved, we would appreciate your comments and would be happy to address them to further improve the paper.

Reviewer_A2r6OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CIRA, a framework that converts expert feedback into reusable experiences to improve natural-language-to-Cypher data synthesis. It organizes feedback by scope, consolidates related experiences and retrieves relevant guidance for future generation without updating model parameters. It also generates more diverse queries and transfers learned guidance across schemas, domains, or generator models, and from Cypher to SQL. Experiments show that CIRA clearly improves pair validity over five feedback rounds and outperforms raw-feedback retrieval.

Critical strengths
  1. This paper addresses the cost and semantic errors of generating NL-to-Cypher training data. The proposed CIRA converts individual corrections into reusable guidance and controls where each experience applies.
  2. CIRA achieves 92% validity, compared with 67% for raw-feedback retrieval and 21% without memory on the same 24 specifications.
  3. The CIRA framework improves results across new schemas, domains, and generator models, with promising results on NL-to-SQL.
  4. Combining CIRA-generated data with benchmark data achieves the best overall execution accuracy for both evaluated models. Besides, this paper shows reduced expert feedback effort as experience accumulates, which has practical efficiency in real scenarios.
Critical weaknesses
  1. This paper uses only five feedback rounds and a relatively small number of expert-reviewed examples. The reported validity rates may not generalize to larger or more diverse settings.
  2. Each fine-tuning configuration is evaluated in a single run. The performance gains should be verified across multiple random seeds and larger datasets. Besides, the Cypher-to-SQL improvement is promising, but the paired test does not reach conventional statistical significance (p = 0.063).
  3. The evaluation does not fully show the contributions of CIRA’s key designs including scope-aware routing, experience consolidation and retrieval. More ablations and comparisons would help establish which components drive the observed improvements and how much of the gain can be attributed specifically to continual experience learning.
  4. CIRA reduces feedback effort over time, but it still requires expert review. The scalability of this process remains uncertain for more complex domains.
Questions for authors
  1. Can authors evaluate CIRA over more feedback rounds, larger datasets or a wider range of graph schemas? Besides, can authors repeat the downstream fine-tuning experiments with multiple random seeds and report confidence intervals?
  2. The Cypher-to-SQL result is not statistically significant at p=0.05 level. Can authors provide stronger evidence for cross-language transfer?
  3. Which components contribute most to the performance gains? Can authors provide ablation of scope routing, experience consolidation and utility-based retrieval?
  4. Can the authors quantify the total cost of CIRA, including LLM calls, validation and repair, expert review and experience management?
  5. How robust is CIRA to incomplete or noisy expert feedback?
3 comments

We thank the reviewer for the constructive comments and for recognizing the practical value of CIRA. We have added targeted analyses and experiments addressing the concerns below; all numbers will be included in the revised PDF.

W1/Q1. Scale, number of rounds, and repeated runs. The 5-round ESGF study tests whether reusable experience accumulates and expert effort decreases as the bank matures. We will clarify this scope in the revision. We now test transfer to three distinct targets—GCMD, Hetionet, and formula_1 SQL—with no target feedback. We expanded the downstream study to five generator architectures, fixing training configurations to 1,000 pairs (except standalone CYQUARK at 986), including all 500+500 mixtures. CypherBench and CIRA use separate 200-example test slices rather than a merged score. We have not repeated every downstream fine-tuning configuration across random seeds; we will state this limitation and avoid treating small differences as variance-calibrated estimates.

W2/Q2. Cross-language transfer. We agree that the original Cypher-to-SQL result (p=0.063) does not meet the conventional (p<0.05) threshold and will no longer call it statistically significant. We treat it as suggestive evidence. In the revised frozen-memory comparison on formula_1 SQL, Static achieves 14/24 valid queries, Dynamic Cheatsheet 13/24, ACE 14/24, and CIRA 19/24, using the same source feedback and no SQL-target feedback. Thus, although the paired significance result remains borderline, CIRA performs best among the evaluated methods and the observed transfer is consistent with the broader comparison. We will present this as promising rather than definitive statistical evidence.

W3/Q3. What drives the gains? We added direct ablations of scope-aware routing, experience consolidation, and utility-based retrieval. Removing any one reduces ESGF round-4 validity by 4-5 of 24 pairs, showing the gain is not due to one component.

We also added Dynamic Cheatsheet and ACE using the authors' code and the same Gemini 3.7 Flash as CIRA. Each receives identical expert feedback from the 91 CIRA pairs in ESGF rounds 0-3, builds its own memory, freezes it after round 3, and is evaluated without target feedback. On ESGF round 4, Static scores 5/24, raw-feedback retrieval 16/24, Dynamic Cheatsheet 13/24, ACE 12/24, and CIRA 22/24. On GCMD / Hetionet / formula_1, CIRA achieves 22/22/19 valid queries, versus 13/16/13 for Dynamic Cheatsheet and 15/17/14 for ACE. These results distinguish CIRA from generic cumulative memory: scope control, consolidation, and utility weighting make feedback reusable, especially across domains.

Variantvalid/24wrong queries
Full CIRA220
No experience management (all feedback compiled, no update/merge/reject)183
No utility-weighted retrieval173
No scope routing (all rules offered everywhere)183

Each component removal costs 4-5 valid pairs and reintroduces query errors the full system had eliminated; checkpoints show that the gain accumulates over rounds. Each arm requires 24 pairs of expert grading, limiting its size but covering all three requested components.

W4/Q5. Cost and expert-review scalability. We reconstructed every model call in the five-round ESGF study at Gemini 3.7 Flash list rates. Per generated pair, Static uses 2.41 calls/$0.023, raw retrieval 2.45/$0.024, and CIRA 3.97/$0.043. CIRA costs about 1.9x more per generated pair but yields about 2.5x more valid data: 76/115 pairs versus 30/115 for Static. Cost per valid pair is therefore $0.064 for CIRA versus $0.087 for Static.

Costs decline as experience accumulates: CIRA's cost per valid pair falls from $0.154 in round 0 to $0.04 in round 4, corrections from 16 to 2 per round, and expert feedback time from 56sec to 22sec per pair. Expert effort is concentrated during experience acquisition and decreases as the bank matures. With a frozen bank and no expert in the loop, a 1,000-pair production run completed in 2.7h. We will make this distinction between the learning phase and production use explicit.

W5/Q4. Incomplete or noisy feedback. Incorrect or deliberately misleading feedback can attack experience learning, since corrupted guidance may propagate to later tasks. CIRA has two partial safeguards, not a defense: feedback never enters the bank verbatim, each review passes the manager, which adds, updates, merges it with existing experiences or rejects it (13 of 57 ESGF decisions were rejects); and utility tracking down-weights experiences whose use precedes failures, so a harmful rule stops being retrieved instead of propagating. Detecting unreliable feedback at intake is important future work.

Overall, the new analyses provide broader model and schema coverage, direct ablations, comparisons with existing experience-learning methods, explicit cost accounting, and a more careful treatment of cross-language transfer.

Thank you for your positive assessment of our work. We have updated the main paper and appendices to address your concerns, along with those raised by the other reviewers. The updated PDF has been uploaded, with new additions highlighted in blue for ease of reference. We would welcome your feedback on these changes. If any of your original concerns remain unresolved, we would appreciate your comments and would be happy to address them to further improve the paper.

Thank you for the detailed rebuttal. I have carefully read all your responses, and most of my concerns have been properly addressed. Based on these results, I retain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.