- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes CIRA, a continual experience-learning framework for synthesizing natural-language-to-Cypher (NL-to-Cypher) training data. CIRA converts human expert corrections into structured, reusable experiences, which are consolidated into an evolving memory bank and selectively retrieved to guide subsequent data generation without updating model parameters. The authors evaluate the framework through iterative feedback learning, cross-schema and cross-domain transfer, and downstream fine-tuning experiments. The results show substantial improvements in expert-judged data validity over static generation and raw-feedback retrieval baselines. The authors further claim that CIRA-generated data improves structural diversity and complements existing benchmark data for downstream Text-to-Cypher training.
Critical strengths
The paper is well motivated by the limitations of existing NL-to-Cypher datasets, particularly their semantic errors and insufficient coverage of complex graph queries. The observation that valuable expert corrections are often discarded rather than reused is also compelling. The proposed CIRA framework provides a reasonable approach to addressing these issues through continual experience learning. The experiments show substantial improvements in data validity, while the cross-schema and cross-domain results suggest promising transferability.
Critical weaknesses
-
Neither the main paper nor the supplementary material provides a complete case study showing how expert feedback is transformed into reusable experiences and applied to new tasks. Without examining actual experience entries and their reuse, it remains unclear whether CIRA learns generalizable principles or retains task-specific information, potentially including answer-related hints. This is a significant limitation given that experience learning is the paper's central contribution.
-
The downstream experiments rely on relatively outdated models (Llama-3.1-8B and Qwen2.5-Coder-1.5B), without evaluating modern coding models such as Qwen3 or stronger frontier models. The GPT version used in the transfer experiment is also unspecified. More importantly, CIRA-generated data fails to improve execution accuracy on the independent CB-politics benchmark, even when combined with the original training data. The reported overall gains depend on including CIRA-generated test examples, weakening the evidence for genuine generalization.
Questions for authors
-
Experience learning: Could the authors provide a complete case study showing the original expert feedback, compiled experience, and its successful reuse? How do they rule out the memorization of task-specific or answer-related information?
-
Downstream generalization: Why does CIRA-generated data fail to improve performance on the independent CB-politics benchmark? Can the authors provide stronger evidence of generalization under matched training budgets?
-
Model selection: Why were more recent and stronger models, such as Qwen3-based 7B/8B coding models or frontier models, not evaluated? What exact GPT model/version was used in the transfer experiment?
3 comments
Follow-up on Rebuttal and Score Update
Thank you for the honest response. I appreciate that you directly addressed my concerns, re-ran the full matrix of training setups, and appropriately narrowed your claims. The new end-to-end case study also clarifies how experiences are abstracted and reused.
Based on these substantial updates, I have raised my rating from 1 to 2. I would consider raising my score further if you can address two remaining points in the revised manuscript:
-
Please integrate all the new results into the revised PDF, and polish the figures and overall presentation for better readability.
-
While I understand the template overlap issue in CypherBench, the CIRA test slice is still generated by your own pipeline. If you could evaluate the fine-tuned models on even a small set of 20 to 50 complex graph queries independently outside the CIRA pipeline, it would provide compelling evidence for downstream generalization.
Thank you for your genuine engagement with our work and for your constructive suggestions. We have addressed both remaining points and uploaded the revised manuscript.
-
We incorporated all new experimental results and improved the figures and overall presentation.
-
We added an independent generalization experiment using 197 complex queries from Mind the Query, spanning four graphs unseen during training. These queries were not generated by CIRA, allowing us to assess generalization beyond CIRA-generated data. The setup and results are reported in Section 4.2 and Table 10.
We appreciate your feedback, which has helped strengthen our evaluation and presentation.
We thank the reviewer. We revised the paper to make the evidence explicit.
W1/Q1. What is actually stored and reused? We add an end-to-end case study showing (i) the original specification and generated query, (ii) expert correction, (iii) compiled experience and applicability scope, and (iv) successful reuse on a later task. CIRA does not store the corrected query as a reusable answer. It stores a generalized rule with conditions governing when the rule applies. The reuse task differs in schema entities and query instance; its success requires applying the learned principle rather than reproducing the source answer.
Effect. Pathfinding valid, rounds 0–4: 0/3, 0/3, 1/2, 3/3, 2/3. Retrieved by 17 later ESGF pairs (11 valid); fired on 9 GCMD pairs (8 valid) and 14 Hetionet pairs (13 valid); never offered on SQL (eligibility filter). This shows cross-schema reuse and scope control.
Consolidation. Experiences are updated, merged, or rejected, not stored verbatim. Of 57 manager decisions: 32 add, 11 update, 1 merge, 13 reject. Reject example: “an exact duplicate of existing rule 'Select constraints for non-trivial aggregation'”. Update-then-merge example: ESGF-anchored “Avoid isolated City geocoder lookups” widened once, then merged into “Connect peripheral entities to core domain concepts” and retired. This retains reusable but scoped guidance.
W2/Q2–Q3. Downstream generalization and model selection. The reviewer is right that the original merged “overall” score did not isolate generalization. We redid the study: (i) training sets are 1,000 pairs, including all 500+500 mixtures (CYQUARK alone has 986); (ii) CypherBench (CB) and CIRA are evaluated on separate 200-example test slices, never merged; (iii) we added three newer architectures (Qwen3.5-9B, Qwen3.5-2B, Gemma-4-E4B) alongside Llama-3.1-8B and Qwen2.5-Coder-1.5B, plus two public synthesis pipelines, CYQUARK and Mind the Query (MtQ), run on the same graph.
Execution accuracy on unseen politics (CB slice / CIRA slice):
The equal-size mixtures address the training-budget concern: gains are not simply from adding more data. On the independent CB-politics slice, CIRA training does not outperform CB training. However, CB is instantiated from only 12 graph-independent templates, and 91% of politics test questions share their exact match-pattern template with the soccer training pool. This overlap creates a strong in-distribution advantage for CB-trained data and limits the slice as a test of broader generalization.
CIRA is instead the strongest single source on the human-style slice across all five models and outperforms CYQUARK and MtQ on both slices. We therefore narrow our claim: CIRA provides complementary, human-style coverage, not a replacement for CB on its own templated distribution. The newer architectures address model selection; we will also specify the exact GPT model/version in the transfer experiment.