acceptodds
Under review as a conference paper at ICLR 2027

Context-Grounded Data Curation and Gate-First RL Reward Design for SQL Autocomplete

Abstract

SQL autocomplete models trained on user editing logs suffer from a supervision mismatch: the recorded completion often depends on information absent at the cursor, such as a table not yet referenced, a literal not in the session, or a clause whose intent emerges only with further typing. We address three coupled problems: what constitutes a valid training target, how to measure both suggestion quality and firing-abstention behavior, and how to train the model to balance useful completions with calibrated silence. Raw editing logs contain content the model could not have inferred at the cursor; we enforce context grounding by trimming each SFT target to its longest inferable prefix and retaining empty targets as silence labels, raising precision by nearly 13 percentage points. To further improve suggestion quality and calibrate the firing decision, we apply RL with deterministic abstention and grounding gates before a graded LLM-judge reward, which prevents the silent collapse observed with a judge-only reward and raises pass@1 on predictable triggers from 35.8% to 43.4%. Since public benchmarks do not score silence, we reconstruct intended completions from real editing sessions and design a selective evaluation framework that jointly measures suggestion quality and firing-abstention behavior. In matched seven-day deployment windows, the 4B dense model, trained on no customer data, raises live acceptance from 17.8% to 26.35% over the previous 30B-A3B MoE while reducing median latency by 71%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SQL autocomplete, context grounding, reinforcement learning, data curation, calibrated abstention, fill-in-the-middle, GRPO, reward design

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kMoSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies supervision mismatch in SQL autocomplete, where recorded edits may depend on information unavailable at the cursor, and discusses how to balance useful suggestions with calibrated silence. The authors trim SFT targets to their longest context-inferable prefixes, retain empty targets as silence labels, and apply RL with deterministic abstention and grounding gates before a graded LLM-judge reward, alongside an evaluation framework reconstructed from real editing sessions. Offline experiments show a 12.9-percentage-point precision gain from context-grounded supervision and an increase in suggest pass@1 from 35.8% to 43.4% with gated RL.

Critical strengths
  1. The paper addresses supervision mismatch through context-grounded targets and explicit silence labels. It distinguishes edits observed later in a session from completions inferable at the cursor, and applies this distinction across SFT and RL. With the backbone and training recipe fixed, replacing raw-edit targets with context-grounded references increases precision from 21.0% to 33.9%.

  2. The experimental evaluation combines broad baseline comparisons, detailed ablations, and real-world deployment evidence. Comparisons include public code and general-purpose models, the previous 30B-A3B production system, and a controlled progression from the 4B base model through SFT to GRPO and SAO, while ablations examine target construction and reward design. Deployment results and recorded user behavior further support practical relevance, while appropriately distinguishing acceptance from offline correctness.

Critical weaknesses
  1. Generalization beyond the evaluated SQL dialect and product surface remains untested. The specific SQL dialect and data platform are anonymized, and the reported results do not establish that the improvements extend to other SQL dialects, such as those used by PostgreSQL or SQLite. Evaluation across additional dialects would strengthen this evidence.

  2. The description of Stage 2 provides insufficient detail on how the context-grounding trim point is determined. Appendix B defines the predictability categories, and Appendix D states that trimming stops at the first unsupported item, but the criteria for deciding which content is inferable from the available context remain unclear. More detailed decision rules and examples distinguishing fully predictable, partially predictable, and unpredictable edits would clarify how the “longest context-inferable prefix” is selected.

Questions for authors
  1. What evidence supports the LLM reward judge’s individual criteria and weighting choices, beyond the overall comparison with SQL-BLEU? Appendix E analyzes the false-fire penalty and squared score, but does not separately examine the doubled semantic-correctness weight or the intent range of 0/1/5. Additional criterion ablations, examples of disagreement with SQL-BLEU, or independent validation of the reward judge would further clarify the contribution of this scoring design.

  2. What distinguishes a rating-1 “safe/partial” completion from a rating-2 “fully useful” completion when both provide only part of the intended edit? The paper appropriately allows useful progress without completing the entire query, making this distinction important for interpreting pass@1. Contrasting examples would help clarify how the evaluator applies these ratings.

  3. Could the authors discuss the practical usefulness of completions with a correct opening followed by a minor error? Appendix C assigns rating 0 to a correct opening followed by unsupported or contradictory content, although users might still benefit from accepting some such completions and making a small correction. Examples or editing-effort analysis could clarify the trade-off between these outputs and shorter, fully correct completions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kMoS

Thank you for the detailed and valuable feedback. Our responses follow; changes are marked in red in the revised PDF.

W2: How is the trim point chosen? An LLM applies a fixed rubric:

  1. Keep SQL syntax, identifiers that appear anywhere in the context at the cursor (prefix, suffix, DDL, history, worksheet), literals that appear in that context, and universal constants (0, 1, NULL, CURRENT_DATE).
  2. Cut at the token boundary before the first unsupported identifier, literal or new alias.
  3. Drop clause jumps (e.g., WHERE → GROUP BY) that the session history does not support.

Synthetic examples (| marks the cursor), with orders(order_id, region) in the DDL:

  • SELECT order_id, reg| + ion FROM orders is kept whole (fully predictable).
  • WHERE region| + = 'APAC', with 'APAC' unseen, is cut to = (partially predictable).
  • WHERE | + customer_tier = 3, with customer_tier absent, becomes empty (unpredictable).

These rules, with examples, are now in App. D.

Q1: Evidence for the judge's criteria and weights. We ran two new analyses (App. E).

Offline ranking (no retraining). On 6,439 gate-passing completions, we measured how well each scoring variant ranks them against three signals the reward judge never sees (AUC, each on the subset where it is available):

ScoreQwen eval. rating 2Claude rating 2Real user accept
Deployed judge0.6880.6980.592
Semantic weight 1, or intent 0/1/2same (Δ ≤ 0.001)samesame
Without the intent criterion0.6210.6240.569
SQL-BLEU0.5940.6000.550

Intent alignment carries the ranking signal; changing the semantic weight or the intent range changes AUC by at most 0.001.

Retraining. We retrained SAO with each variant under the Table 8 protocol:

RewardPrec.S@1D@1Bl@1
Deployed (semantic weight 2, intent 0/1/5)33.637.277.860.6
Intent 0/1/232.936.078.060.2
Semantic weight 133.036.677.860.4

All differences are within about one point, so the results are robust to these choices. The deployed weights are slightly better on precision, S@1 and Bl@1, so we keep them as the default.

Versus SQL-BLEU: BLEU scores below 0.3 on 43% of correct (rating-2) completions, mostly short correct prefixes penalized by its brevity term; the judge does so on 7%.

Q2/Q3: Rating 1 vs. 2, and a correct opening followed by a minor error. A correct opening followed by a grounded, non-contradictory deviation already gets rating 1 (partial credit). After a correct opening, an unsupported or contradictory continuation receives 0. At a normal cursor, rating 2 means everything emitted is correct for the user's intended edit; a short, correct prefix is enough. Example, with 'EMEA' and the column ds in the context: the user later executed WHERE region = 'EMEA' AND ds >= …, and the cursor is after WHERE region:

  • = 'EMEA' → 2
  • = 'EMEA' AND ds IS NOT NULL → 1
  • = 'APAC' (not in context) → 0

Real users follow this split: acceptance is 9% / 15% / 52% for ratings 0 / 1 / 2 (Table 9). Rating-0 completions are rarely accepted, while short, fully correct ones are accepted most often, so pass@1 counts only rating 2, while mean r gives rating 1 partial credit. The examples are now in App. C.

W1: Other dialects. Porting mainly requires three changes: the SQL parser dialect used by the grounding gate, the list of built-ins treated as universal constants, and the prompts. The reconstruction, trimming and gate-first reward stay the same. Evaluating another dialect needs editing logs from it, which we leave to future work. App. A now describes this.

I thank the authors for clarifying how the context-grounding trim point is selected and how ratings 1 and 2 are distinguished. The two additional analyses of the LLM judge also addressed my questions, and I have no remaining concerns about the LLM judge. Although transfer to other SQL dialects has not yet been empirically demonstrated, the authors' discussion and proposed dialect-porting plan provide a reasonable path forward. Overall, my main concerns are all addressed. I appreciate the authors' detailed explanations, additional analyses, and supporting statistics.

Thank you so much for your reply and for taking the time to review our additional analyses. We are delighted that our responses addressed your main concerns, and we will definitely incorporate these discussions and refinements into the next version of our manuscript.

Reviewer_TCHTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the problem of super vision mismatch in SQL autocomplete models trained on user editing logs. This problem arises because the recorded completion can depends on information not available at the cursor, for example, a table is not yet referenced or it is not possible to gauge the intent at the current state of the clause typed. This problem can arise because raw editing logs contain content that could not have been inferred by the model at the cursor. The authors enforce context grounding by trimming each SFT target to its longest inferable prefix and retaining empty targets as silence labels. This raises precision by nearly 13 points. They further apply RL with deterministic abstention and grounding gates before a graded LLM-judge reward, preventing silent collapes and and raising pass@1 on predictable triggers from 35.8% to 43.4%. In a seven day deployment window, this raised live acceptance from 17.8% to 26.35% and reduced median latency by 71%.

Critical strengths

The paper studies a relevant problem in current day AI platforms. The writing is very clear. The problem studied is very well stated and the methods are clearly described for the most part. The evaluation is also clearly outlined and seems to follow principled measures. The resultant model improvements result in substantial improvement in quality and latency in live deployment over the existing model in their production system.

Critical weaknesses

The baselines do not include CodeLlama 34B or DeepSeek-Coder-7B, which seem to be reasonable off the shelf models to compare with. Other than that, in general I don't see any major weaknesses to be flagged in the paper.

Questions for authors

In the two stage reference construction, you mention that Stage 2 keeps the longest prefix of a matched raw edit that is context-grounded by the prefix, suffix, schema, or session history. Is this determined through deterministic algorithms or by using an LLM?

Why were two standard off the shelf models like CodeLlama 34B or DeepSeek-Coder-7B not included among the baselines?

Is there an intuitive reason why the Qwen3-4B-Instruct-2507 model reduced by 5% in D@1 after context grounding or gated reward RL (GRPO and SAO)?

1 comment
AuthorsOpenReview ↗

Response to Reviewer TCHT

Thank you for the positive and constructive feedback. Our responses follow; changes are marked in red in the revised PDF.

W1/Q2: CodeLlama-34B and DeepSeek-Coder baselines. We added both under the Table 1 protocol: the same instructions (including the rule to return nothing when unsure), the same references and the same evaluator, with each model in its own chat format.

ModelPrec.S@1D@1Decline rate
CodeLlama-34B-Instruct4.510.00.50.0
DeepSeek-Coder-6.7B-Instruct3.77.910.36.3
Qwen3.6-35B-A3B (best public, Table 1)11.814.555.843.9
Ours (4B + SFT + GRPO)38.641.079.550.8

Both almost never stay silent, despite the instruction. Both rows are now in Table 1, with interface details in App. K.

Q1: Is Stage 2 deterministic or LLM-based? LLM-based. An LLM picks the trim point following a fixed rubric: keep SQL syntax, universal constants, and identifiers and literals that appear in the context at the cursor; cut before the first item that does not. For training data, the LLM also decides which empty targets to keep as silence labels; deterministic code then only keeps rows that were matched and marked as usable for training. Sec. 2.1.1 and App. D now state this explicitly.

Q3: Why does D@1 drop after training? D@1 is pass@1 on requests whose reference is empty: an empty output passes, as does a rating-2 completion. It drops by about 7 points (86.9 for the base model, 80.1 / 79.5 / 80.2 for SFT / GRPO / SAO), which reflects a trade-off between firing and staying silent. The base model gets its high D@1 by being overly conservative: it returns empty on 86% of samples, including 84% of fully predictable cursors, so its S@1 is only 0.5, which makes it of little practical use as an autocomplete model. After training, the model completes where the context supports it, at the cost of sometimes firing where it should stay silent. Overall, from base to SAO, S@1 rises from 0.5 to 43.4 and blended pass@1 from 50.4 to 64.7. RL itself keeps D@1 stable (79.5–80.2).

Reviewer_gygtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a supervision mismatch in SQL autocomplete trained from editing logs: the edit eventually made by a user may depend on information unavailable at the earlier cursor position. The authors address this mismatch through context-grounded target construction, retaining the longest supported prefix and explicit empty targets, and through a gate-first RL reward that handles false firing, unnecessary abstention, and grounding violations before applying a graded LLM-judge reward. The paper also introduces an evaluation framework that jointly measures completion quality and firing behavior.

Critical strengths
  1. The distinction between an eventual user edit and a completion inferable at the cursor is important for realistic autocomplete and is poorly captured by conventional completion benchmarks. The paper operationalizes this distinction rather than treating it only as a conceptual observation.
  2. The paper evaluates both output quality and firing decisions, and its matched-firing-rate confidence baselines help rule out a simple explanation based solely on increased abstention. The same-backbone comparisons, paired confidence intervals, and deployment results make this a useful empirical contribution, even though the individual optimization and gating components are relatively simple.
Critical weaknesses
  1. The training-reference construction pipeline and the offline evaluator both use Qwen3.6-27B-FP8. This creates a risk that grounded SFT preferentially aligns with the same model’s judgments of contextual support and completion usefulness. The paper provides useful checks, including agreement with Claude, manual reference checks, and a correlation between evaluator ratings and user acceptance. However, these do not fully rule out correlated biases or directly validate output quality across the compared systems.
  2. The "4B beats 30B" claim and the deployment gains are confounded. The previous 30B-A3B system was trained mostly on synthetic data. Its gap to the new 4B model therefore mixes three factors: real-session data, context grounding, and model choice. Notably, the 4B model trained on raw-edit targets reaches 21.0% precision, essentially matching the previous system's 22.0%. This suggests the gain comes from the data pipeline rather than from anything specific to the 4B model. Without a 30B-A3B model trained on the same grounded data, the TL;DR's framing that a 4B model beats a 30B MoE is potentially misleading.
Questions for authors
  1. Since Qwen3.6-27B-FP8 both generates the training references (including the grounding trim) and serves as the offline evaluator, can the main conclusions of Table 1, Fig. 3, and Table 7 be reproduced with Claude Sonnet 4.6 or with human ratings as the evaluator?

  2. Is the Stage 2 trim on the training set performed by the LLM, by deterministic rules, or by both? Appendix D says SFT trimming "checks both identifiers and literals and cuts a raw edit at the first unsupported item," while Sec. 2.1.1 describes the trim as LLM-assisted.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gygt

Thank you for the constructive and valuable feedback. Our responses follow; changes are marked in red in the revised PDF.

W1/Q1: Qwen builds training targets and evaluates. For context, the evaluation references are not built by Qwen: both frozen evaluation sets were reconstructed with Claude Sonnet 4.6 (App. K). Qwen3.6-27B-FP8 builds only the training targets. To test the remaining coupling, we re-scored the same generations with Claude Sonnet 4.6 as the evaluator, using the same references:

Shown-precision (%)Qwen evaluatorClaude evaluator
Previous 30B-A3B22.023.5
Raw-edit SFT (4B)21.022.7
Grounded SFT (4B)33.937.4
+ GRPO38.640.4
+ SAO40.643.2

The main conclusions hold under Claude. The gain of grounded over raw-edit targets is +12.9 points under Qwen and +14.7 under Claude. The Fig. 3 result also holds: trim-only targets drop decline pass@1 to 5.5%, and adding silence labels brings it back to 80.8%. So does Table 7: among nonempty outputs on partially predictable requests, the share with any grounding violation falls from 71.6% to 27.4% under Qwen and from 71.4% to 28.8% under Claude. Details are in the new Table R2 (App. H).

W2: "4B beats 30B" is confounded. We agree and have reframed this claim. Sec. 3.5 is a deployment comparison, not a controlled one: the two systems differ in data, targets and model. It shows that our data and training pipeline produces a better specialized model than the previous production pipeline. The controlled evidence is the same-backbone comparisons in Secs. 3.2–3.4: on the same 4B model, grounded targets raise offline precision from 21.0% (raw edits) to 33.9%, and RL raises it to 38.6%. The raw-edit 4B model is close to the previous system (21.0% vs. 22.0%), consistent with the reviewer's reading that the gain comes from the data pipeline. We have revised the abstract and Sec. 3.5 accordingly. The TL;DR cannot be edited during the rebuttal; we will change it to: "Context-grounded SFT targets and gated RL rewards produce a specialized 4B SQL autocomplete model that replaced the previous production model with higher precision, higher acceptance, and lower latency."

Q2: Is the Stage-2 trim done by an LLM or by rules? The trim point is chosen by an LLM following a fixed rubric (Claude for evaluation references, Qwen for training targets). App. D describes that rubric, so the two passages describe the same step. The rubric keeps SQL syntax, universal constants, and identifiers and literals that appear in the context at the cursor, and cuts before the first item that does not. For training data, the LLM also decides which empty targets to keep as silence labels; deterministic code then only keeps rows that were matched and marked as usable for training. Sec. 2.1.1 and App. D now state this explicitly.

I believe the authors' response has addressed my concerns. I will maintain my original score. Good luck.

Thank you so much for your reply and kind wishes. We are delighted that our response addressed your concerns, and we truly appreciate your constructive feedback throughout the review process. We will make sure all of these points and refinements are cleanly integrated into the next version of our manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF in which all changes are marked in red. The main changes are:

  1. Evaluator robustness (gygt). We re-scored the same generations with Claude Sonnet 4.6 as the offline evaluator. The main conclusions of Table 1, Fig. 3 and Table 7 hold (new Table R2 in App. H; Sec. 3.1).
  2. Deployment comparison reframed (gygt). Sec. 3.5 and the abstract now separate the deployment comparison from the controlled same-backbone evidence in Secs. 3.2–3.4.
  3. Stage-2 trim clarified (gygt, TCHT, kMoS). Sec. 2.1.1 and App. D now state that an LLM picks the trim point with a fixed rubric, and App. D gives the decision rules with examples.
  4. New baselines (TCHT). CodeLlama-34B-Instruct and DeepSeek-Coder-6.7B-Instruct are added to Table 1, with interface details in App. K.
  5. Judge criteria and weights (kMoS). App. E adds an offline ranking analysis of the scoring variants (Table R1) and a retraining ablation of the semantic weight and the intent range (Table R3).
  6. Rating examples (kMoS). App. C adds contrasting examples for ratings 0, 1 and 2.
  7. Other dialects (kMoS). App. A describes what porting to another SQL dialect requires.
  8. Correction. Table 12 now lists the correct number of SFT epochs (2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.