acceptodds
Under review as a conference paper at ICLR 2027

CURING THE SILENT ROT: DIAGNOSING AND STRESS- TESTING CONTEXT BANKS FOR TEXT-TO-SQL

Abstract

Text-to-SQL systems increasingly rely on context banks: curated collections of example queries, metric definitions, and domain knowledge that tell a model how an organization writes its queries. In the enterprise, a bank grows over years as many analysts add entries, checked one at a time for correctness (does the SQL answer its own question?) and rarely for consistency with one another. Yet cor- rectness is often loosely defined: a question seldom fixes how an entity is counted or which table defines it, so two entries can each pass review yet resolve the same decision in opposite ways. We call this condition context bank rot. A second ques- tion follows: does a system that uses a bank generalize from it, or only reproduce it? Existing benchmarks cannot tell, as they do not control how far a test question sits from the context. We answer both questions by comparing queries decision by decision, with a fast scorer distilled from human-calibrated LLM judges. Turned inward, the comparison diagnoses and repairs a bank. Fine-tuning on a cleaned BIRD-train bank beats a same-size random subset at every epoch (+2.6 points on average), and 92.4% of our removals had their SQL corrected by an independent expert audit, against 45.6% of audited entries. In-context examples that follow one convention beat those following the opposite one by 13.3 points, while a mix is barely better than none. Turned outward, the comparison generates test ques- tions at a controlled distance from a bank, a pipeline we call CAT (Context-Aware Testing). On CAT-BIRD, for retrieval and fine-tuning alike, the benefit of a bank falls step by step with distance and is largely gone once about three quarters of a question’s logic is new, and on CAT-Spider the benefit of retrieval likewise shrinks with distance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Text-to-SQL, LLMs for databases, Context improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zaXZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies cross-entry consistency in Text-to-SQL context banks and model generalization beyond the bank. The authors decompose SQL queries into semantic decisions, use two LLM judges to generate alignment, conflict, and novelty labels, and distill these labels into a pairwise scorer. They further propose a Bank Health Index, a two-track cleaning protocol, and CAT benchmarks organized by semantic distance. The experiments examine the effects of cleaning on fine-tuning, example conventions on inference, and increasing question distance on the benefit of context.

Critical strengths
  1. The focus on cross-entry convention consistency has practical value. The paper distinguishes whether each SQL query answers its own question from whether different examples follow the same conventions. It compares matched, opposite, mixed, and no-example conditions. Table 3 reports a 13.3-percentage-point accuracy gap between matched and opposite conventions. The experiment on naturally grouped questions additionally compares cases where the wording specifies the convention with cases where it does not, providing evidence that examples influence model behavior by resolving unspecified decisions.
  2. The cleaning experiments include meaningful controls. Table 2 compares the cleaned training set with an equally sized random subset stratified by database and SQL length, with improvements at every reported epoch. The cross-check against expert corrections in BIRD-Platinum also supports the presence of genuine annotation problems among the removed entries. The authors explicitly acknowledge that this experiment evaluates cleaning as a whole and does not isolate the contribution of cross-entry consistency.
Critical weaknesses
  1. Key semantic judgment procedures lack complete reproducibility details.

    The method relies on LLM judgments for semantic-unit grounding, ACN labeling, and cleaning decisions, but the complete prompts, output schemas, and post-processing implementation are not provided. The supplied PDF contains no accessible code link. Since the judge protocol was refined over 14 calibration rounds, these missing details are important for reproducing the labels and cleaning results.

  2. The overall workflow is insufficiently clear.

    The connections among annotation, scorer training, bank cleaning, and CAT construction are difficult to follow from the text alone. An overall framework diagram showing the data flow, separating training from application, and identifying the roles of the SQL parser, scorer, and LLM judges would make the method easier to understand.

Questions for authors
  1. Could you provide the implementation materials needed to reproduce the core pipeline? Could you share anonymous code and the final prompts, output schemas, and post-processing rules for the core pipeline? A worked example showing how semantic units are matched and converted into an ACN label would also help.
  2. What are the actual token usage and end-to-end computational costs? Please report per-pair teacher token usage, total student training tokens, and the end-to-end cost and runtime of bank cleaning and CAT construction. How much does distillation save compared with directly using LLM judges?
  3. Please consider expanding the ethics statement to describe human expert participation. Appendix C.1 mentions expert participation in 14 calibration rounds. Briefly describing the expert's background, involvement, and workload, and, where applicable, payment or compensation arrangements, would improve transparency about the human calibration process and make the ethics statement more complete.
2 comments
AuthorsOpenReview ↗

Our Response to Reviewer zaXZ

We thank Reviewer zaXZ for the constructive feedback on reproducibility, workflow clarity, compute reporting, and ethics documentation; we have addressed all four points in the revised paper and supplementary archive.

Reproducibility materials, prompts, schemas, and worked example (Weakness 1 and Question 1)

We agree completely and have uploaded an anonymized supplementary archive containing the full pipeline code, verbatim prompts (for unit grounding, ACN labeling, Track A auditing, Track B adjudication, and CAT probe generation and verification), exact JSON output schemas for every LLM call, the deterministic post processing code that turns a judge's unit map into an ACN profile, the complete convention rulebook with conditions, exceptions, and SQL examples, and the scorer's input format and training code. In the revised paper, Appendix K also provides the verbatim prompts alongside a complete worked example taking one real pair through every stage: unit extraction, span assignment, role matching, per role scoring, and the final ACN profile.

End to end workflow diagram (Weakness 2)

Thank you for this suggestion. We added an overview diagram in Figure 2 (caption in red) tracing the data flow from end to end and separating one time training from application. Panel 1 shows scorer creation, which runs once: a SQL parser extracts decision units, LLM judges label pairs under human calibrated rules, and their consensus labels join exact programmatic variants to train the scorer. Panels 2 and 3 apply the frozen scorer, the only component that runs over all pairs: in Panel 2 its outputs compute the Bank Health Index directly and select candidate pairs for Track B rulebook adjudication alongside linear Track A auditing, while in Panel 3 it keeps CAT probes that fall in their target distance band with no conflict before LLM SQL verification. Every box is color coded by whether the work is done by deterministic code, the local scorer, an LLM call, or a human.

Token usage, runtime, and distillation savings (Question 2)

Appendix B now compares the full compute and token cost of the local scorer against direct LLM judging. Training the scorer is a one time job taking 30.8 hours on a single 80GB GPU. At inference it runs locally at 5 to 9 pairs per second, so 100,000 pairs take roughly 3 to 6 GPU hours. Because BIRD-dev alone has more than 100,000 same database pairs, auditing a bank without distillation would require over 200,000 frontier LLM calls on BIRD-dev at every audit. With the local scorer, only flagged pairs go to the LLM judges: just 2,127 pairs on BIRD-dev (about 2% of pairs) and 3,101 on BIRD-train (Table 10), cutting pairwise frontier LLM calls by roughly 98%.

Human expert participation in the Ethics Statement (Question 3)

Thank you for raising this. We expanded the Ethics Statement in the revised paper (in red) to describe human participation clearly: both the expert who adjudicated the 14 calibration rounds (Appendix C.1) and the three blind annotators in our new validation study (Appendix J) are authors of this paper working on Text-to-SQL. No external or crowd annotators were recruited, so no compensation was involved, and all annotated material comes strictly from public benchmark questions and SQL.

Thank you to the authors for their responses and additional experiments. I believe some of my experimental concerns have been addressed, but I remain largely unconvinced about the idea.

Moreover, the revisions to the PDF are extensive (and cover more than half of the original main draft), and I don’t have the bandwidth to fully re-evaluate the revised manuscript and all the additional experiments during the rebuttal period. Given the substantial presentation issues in the original submission, I believe the paper would benefit from a resubmission with a clearer focus on contributions and experimental evidence. Hence, I will maintain my current score.

Reviewer_yCcnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper defines a problem in Text-toSQL systems, where in enterprise settings, new entries in context banks are only checked one at a time for correctness, but not for consistency with each other, or generalization. The condition where two entries resolve the same question in different ways is being called as context bank rot. To check for consistency, each entry is compared against other entries, by decomposing SQL into what is defined as "semantic units" but is in reality synactic units (aggregation, filter, etc) and LLM judges are used to compare against pairs and compute align-conflict-novel (ACN) for the pair. Aggregation of this gives Bank Health Index (BHI). Using these, and two tracks (one for checking per-entry correctness, and one for checking cross-entry consistency), the data is cleaned and experiments are run to compare by finetuning on clean vs uncleaned, and gains are reported for uncleaned, matched, and opposite convention, although based on the tables, most of the cleaning seems to be coming from track A which is the per entry correctness track. Track B (cross-entry consistency) contributes very less to the cleaning, and therefore most of the accuracy gains could just be from per entry cleaning, which is already established in the domain, and the main claims are not sufficiently supported by experiments. The alignment scores are used in a pipeline CAT, which grows a clean bank with new questions at a chosen distance from ACN profile. Retrieval and finetuning are found to decay with distance.

Minor: There is a discrepancy in the title on openreview and title on paper, this does not affect my assessment, but wanted to flag for consistency

Critical strengths
  1. Most data noise related prior work do focus on per entry correctness rather than data bank level consistency, so the concept is useful and the drift exists in enterprise settings where new entries are rarely checked for consistency with existing ones
  2. The score computed for measuring distance being reused for generating more data is interesting
  3. The problem definition and experimental details are elaborated and defined, with some of the functional definitions being useful for future work, like Bank Health Index, and ACN and concept of CAT, and even though the "semantic unit" is not really semantic, it is a good grouping of syntactically similar samples
Critical weaknesses
  1. The novel components (ACN/BHI/cross-entry checking) contribute very little to the repair. More than 90% of the removals or cleaning seem to come from Track A which is the per entry correctness check for whether a SQL and the question are matching and correct. ACN is only used in track B where pairwise consistency is considered, and track B contributes less than 10% to the cleaning. The improvement in the finetuning experiments then mainly comes from the per-entry correctness and not cross entry consistency
  2. The "semantic units" defined are syntactic, and they are defined by the SQL syntax (aggregations, filter etc). The CAT distance is also syntactic, and a question can share fewer clauses with bank while still maintaining convention. The less that a question shares with a bank, the less it will help with, so a loss in benefit is expected here
  3. Although the positioning is identified as context bank rot forming due to eventual drift by adding new data that does not stay consistent with overall bank level agreement, pairwise comparisons are done rather than measuring bank-level consistency. Pairwise can show that two samples disagree with each other but does not confirm which of them is right or more prevalent. Distribution level metrics would be more accurate
  4. The pipeline heavily relies on LLMs which act as judges, auditors, generators, verifiers, and synthetic generators. Since the main contribution is measurement, a held out human annotated sample set would solidify contributions more.
Questions for authors
  1. How many of the conflicts involve the same concept realized through different schema paths (like “eligible for loans” example)? and how well do the scorer and judges detect such cases? For example, what is number of constructed pairs detected that define one concept through different tables?
  2. Are experiments done on gains from Track A alone and Track B alone, with the same size matched random controls? If finetuning gains are mostly affected by Track A, then per-entry correctness would be the majority contributor to the accuracy gains
2 comments
AuthorsOpenReview ↗

Our Response to yCcn

We thank Reviewer for the thoughtful critique, which motivated additional ablations and human annotation study.

W 1 and Q 2

Thank you for suggesting this ablation: we ran Qwen2.5-7B on Track A alone and Track B alone against size matched random controls (Section 5.1, Appendix I, Table 16). Track B before bug removal gives -0.2 points vs. control (51.0% vs. 51.2%) because 1,078 broken SQL queries remain in raw BIRD-train, whereas clearing bugs first with Track A gains +2.4 (52.9% vs. 50.5%) and running both tracks gains +2.6 (53.8% vs. 51.2%, or +3.0 over the 50.8% uncleaned pool). Comparing the two cleaned sets directly, adding Track B to Track A deletes only 80 remaining conflicts (1.4% of the pool) yet lifts accuracy from 52.9% to 53.8% overall and by 1.2 to 1.4 points across every converged epoch (Epochs 2 to 4). Track A carries the larger share on raw BIRD-train because crowdworker data has far more broken SQL (45.6% fixed in BIRD-Platinum) than convention splits and BIRD-dev's test gold mixes conventions (Table 11), making that lift from 80 removals notable (on BIRD-dev, Track B is half of removals, 78 of 157). Once entries pass single entry review and an organization picks one convention, as in Section 5.2 and Table 12 (and LiveSQLBench's 18.7% conflict rate on analytical SQL, Table 1), Track A removes nothing and consistency drives the full gain: +13.3 points on paired questions, +29.8 on 151 outside questions, and a 75.2% to 58.1% drop as conflicts reach half of a clean bank. ACN also powers the Bank Health Index (0.90 Spearman with accuracy, Table 12) and every distance in CAT (Section 6).

W 2

Point well taken: we renamed semantic units to decision units throughout. While each unit starts from a SQL clause, alignment and conflict depend on the question and schema rather than syntax: in Figure 1b, filtering disp and joining loan share neither table nor clause, yet tying both to "eligible for loans" surfaces the conflict. CAT distance steps down shared SQL decisions to test whether queries sharing few clauses keep bank conventions. Cross database examples in Section 5.2 swing accuracy by 29.8 points, showing conventions transfer without shared clauses when isolated; yet on CAT-BIRD at d3, once three quarters of logic is new, neither retrieval nor 20 epoch fine tuning beats uncovered databases, showing standard RAG and SFT lean on structural overlap rather than carrying conventions into new logic (Section 6).

W 3

We do measure consistency at the bank level: the Bank Health Index aggregates all overlapping pairs into one bank score, and conflict hubs evaluate each entry against all neighbors (Section 4.1). Pairs link shared decisions across tables (loan vs. disp), and neighbor counts show prevalence directly. Still, prevalence cannot say which convention is "right": once single entry checks remove bugs, both sides are valid and cleaning seeks a consistent bank, lifting accuracy under either choice (Table 11). Picking the majority drops fewer entries, while our editable rulebook enforces a chosen policy when logs lean the other way or split evenly as in BIRD.

W 4

When LLMs wear many hats in a measurement paper, the fair worry is that they might grade their own homework. Three authors therefore labeled 143 decision units blind across 25 held out pairs including five thorny edge cases. Humans agreed closely (Fleiss' kappa 0.79), and our two judge consensus matched their majority on 93% of decisions with zero align vs. conflict confusions, sitting within 0.002 on pair conflict and catching all 5 conflict pairs with 0 false alarms. Elsewhere too, no model grades its own homework: BIRD-Platinum experts fixed 92.4% of our removals; SQL syntax matching reproduces the +12.1 point swing on 470 queries; Claude wrote all twins in Table 12 while Gemini took the exam on 246 untouched queries and dropped 17.1 points; and on CAT, Qwen and Gemma wrote zero probes yet show the same decay, scored by SQL execution.

Q 1

By construction, our pipeline is geared to catch cross path conflicts: both the judges and scorer match decisions by their shared role in the questions rather than by table name, and Track B's rulebook explicitly resolves competing join paths and column mappings (R2, R4, R9). When we classify our decided Track B conflicts using two independent LLMs and require both to agree, 17 of 106 on BIRD-dev (16%) and 42 of 683 on BIRD-train (6%) realize one concept across different tables or join paths, as when two Shakespeare queries count a play's characters by joining to characters versus counting character_id on paragraphs. Although our synthetically edited conflict pairs only swap columns within one table, unmodified BIRD-train pairs supply strong cross table supervision: by deterministic SQL check, 477 of the 895 conflicting training pairs (53%) touch different tables, and BIRD-dev cross table conflicts score right alongside same table ones (median 0.060 vs. 0.069).

Thank you for the rebuttal, the updated version of the paper, and the new experiments.

  • Renaming semantic units to decision units reads much better and is less misleading
  • Regarding the ablation for Track A and Track B - "Once entries pass single entry review and an organization picks one convention, as in Section 5.2 and Table 12 (and LiveSQLBench's 18.7% conflict rate on analytical SQL, Table 1), Track A removes nothing and consistency drives the full gain" - Track A also does single entry review, so it seems natural that after single entry review Track A does not remove anything, and does not seem conclusive, and this concern still remains.
  • Thank you for confirming the role of LLM judges and human assessment. This resolves my concern on this point.

Overall, the changes improve readability and add more robustness with human assessment. The core problem is interesting, of context bank rot, but the evidence does not seem to show that cross entry consistency is the main source of the gains. I will raise the presentation score but keep the other scores unchanged.

Reviewer_iCdROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

-two problems in context-aware text-to-SQL are studied in this paper: inconsistency within a context bank (“context bank rot”) and how far systems actually generalize from such a bank. -authors propose semantic-unit comparisons and ACN profiles to detect cross-entry conflicts, a bank cleaning procedure with a Bank Health Index -also, CAT - a benchmark construction pipeline that controls distance between test questions and the bank.

Experiments cover bank diagnosis, cleaning for fine-tuning, in-context convention consistency, and controlled generalization on CAT-BIRD and CAT-Spider.

Critical strengths

-problem formulation is strong and practically relevant. The notion that a bank can be locally correct yet globally inconsistent is real -empirical story is broad and mostly convincing. The paper does not rely on a single benchmark number

Critical weaknesses

-LLM judges sit at the center of the method - expert-audit check helps, but the method still inherits a large amount of judge bias and possible circularity -some CAT evidence is less clean than the headline suggests. The paper itself notes that CAT-Spider d2/d3 miss their targets, that probes are generated by evaluated models, and that each configuration uses one seed. So the generalization claims are strongest on CAT-BIRD and weaker as a general benchmark claim.

Questions for authors

Can you quantify how much the main cleaning and CAT conclusions change under alternative judge ensembles or a substantially smaller human-annotated calibration set?

For CAT, can you separate the effect of generation artifacts from true distance by evaluating on a manually curated subset of probes?

1 comment
AuthorsOpenReview ↗

Our response to reviewer iCdR

We thank Reviewer iCdR for the constructive assessment and for highlighting the practical relevance of our problem formulation and the breadth of our empirical results.

Judge validation, circularity, and calibration size (Weakness 1 and Question 1)

We agree that judge bias is the central risk, so we added a blind human study in Appendix J (Tables 17 to 19). Across 143 decision units in 25 pairs labeled by three authors blinded to system outputs (Fleiss' kappa 0.79), our two judge consensus matches the human majority on 93% of decisions with zero align versus conflict confusions, catching all 5 conflict pairs with no false alarms while Claude alone misses 2 and Gemini alone adds 3 false alarms. Reliance on human calibration is also modest by design: calibration used only 148 expert checked pairs (Appendix B), and a third of scorer training pairs are built by programmatic SQL edits with exact labels. Crucially, there is no downstream circularity because judges never score model outputs: every result is execution accuracy against untouched benchmark gold SQL, backed by two checks that use no LLM judge at all (92.4% expert SQL correction rate on our removals in BIRD-Platinum versus 45.6% overall, and +12.1 points from pure SQL pattern matching on 470 queries in Appendix E.3).

CAT generalization scope, generation artifacts, and run stability (Weakness 2 and Question 2)

On CAT, we agree that our strongest generalization evidence is on CAT-BIRD, and we have updated the abstract, Section 6, and conclusion to scope the main claim there while presenting CAT-Spider as a secondary pipeline instance. Against generation artifacts, Qwen2.5-7B and Gemma-4-12B wrote zero probes yet show the exact same decay as the retrieval models (Qwen's gain over its control drops from +38.0 points at d0 to +15.5, +9.0, and +1.0 across d1 to d3), and flat zero context accuracy across d0 to d3 confirms distant probes are not simply harder. Finally, retrieval runs deterministically at temperature 0, and while fine tuning uses one seed per arm, the distance decay holds steadily across every checkpoint from Epoch 5 to Epoch 20 (Table 15).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.