- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies supervision mismatch in SQL autocomplete, where recorded edits may depend on information unavailable at the cursor, and discusses how to balance useful suggestions with calibrated silence. The authors trim SFT targets to their longest context-inferable prefixes, retain empty targets as silence labels, and apply RL with deterministic abstention and grounding gates before a graded LLM-judge reward, alongside an evaluation framework reconstructed from real editing sessions. Offline experiments show a 12.9-percentage-point precision gain from context-grounded supervision and an increase in suggest pass@1 from 35.8% to 43.4% with gated RL.
Critical strengths
-
The paper addresses supervision mismatch through context-grounded targets and explicit silence labels. It distinguishes edits observed later in a session from completions inferable at the cursor, and applies this distinction across SFT and RL. With the backbone and training recipe fixed, replacing raw-edit targets with context-grounded references increases precision from 21.0% to 33.9%.
-
The experimental evaluation combines broad baseline comparisons, detailed ablations, and real-world deployment evidence. Comparisons include public code and general-purpose models, the previous 30B-A3B production system, and a controlled progression from the 4B base model through SFT to GRPO and SAO, while ablations examine target construction and reward design. Deployment results and recorded user behavior further support practical relevance, while appropriately distinguishing acceptance from offline correctness.
Critical weaknesses
-
Generalization beyond the evaluated SQL dialect and product surface remains untested. The specific SQL dialect and data platform are anonymized, and the reported results do not establish that the improvements extend to other SQL dialects, such as those used by PostgreSQL or SQLite. Evaluation across additional dialects would strengthen this evidence.
-
The description of Stage 2 provides insufficient detail on how the context-grounding trim point is determined. Appendix B defines the predictability categories, and Appendix D states that trimming stops at the first unsupported item, but the criteria for deciding which content is inferable from the available context remain unclear. More detailed decision rules and examples distinguishing fully predictable, partially predictable, and unpredictable edits would clarify how the “longest context-inferable prefix” is selected.
Questions for authors
-
What evidence supports the LLM reward judge’s individual criteria and weighting choices, beyond the overall comparison with SQL-BLEU? Appendix E analyzes the false-fire penalty and squared score, but does not separately examine the doubled semantic-correctness weight or the intent range of 0/1/5. Additional criterion ablations, examples of disagreement with SQL-BLEU, or independent validation of the reward judge would further clarify the contribution of this scoring design.
-
What distinguishes a rating-1 “safe/partial” completion from a rating-2 “fully useful” completion when both provide only part of the intended edit? The paper appropriately allows useful progress without completing the entire query, making this distinction important for interpreting pass@1. Contrasting examples would help clarify how the evaluator applies these ratings.
-
Could the authors discuss the practical usefulness of completions with a correct opening followed by a minor error? Appendix C assigns rating 0 to a correct opening followed by unsupported or contradictory content, although users might still benefit from accepting some such completions and making a small correction. Examples or editing-effort analysis could clarify the trade-off between these outputs and shorter, fully correct completions.
3 comments
I thank the authors for clarifying how the context-grounding trim point is selected and how ratings 1 and 2 are distinguished. The two additional analyses of the LLM judge also addressed my questions, and I have no remaining concerns about the LLM judge. Although transfer to other SQL dialects has not yet been empirically demonstrated, the authors' discussion and proposed dialect-porting plan provide a reasonable path forward. Overall, my main concerns are all addressed. I appreciate the authors' detailed explanations, additional analyses, and supporting statistics.
Thank you so much for your reply and for taking the time to review our additional analyses. We are delighted that our responses addressed your main concerns, and we will definitely incorporate these discussions and refinements into the next version of our manuscript.
Response to Reviewer kMoS
Thank you for the detailed and valuable feedback. Our responses follow; changes are marked in red in the revised PDF.
W2: How is the trim point chosen? An LLM applies a fixed rubric:
Synthetic examples (
|marks the cursor), withorders(order_id, region)in the DDL:SELECT order_id, reg|+ion FROM ordersis kept whole (fully predictable).WHERE region|+= 'APAC', with'APAC'unseen, is cut to=(partially predictable).WHERE |+customer_tier = 3, withcustomer_tierabsent, becomes empty (unpredictable).These rules, with examples, are now in App. D.
Q1: Evidence for the judge's criteria and weights. We ran two new analyses (App. E).
Offline ranking (no retraining). On 6,439 gate-passing completions, we measured how well each scoring variant ranks them against three signals the reward judge never sees (AUC, each on the subset where it is available):
Intent alignment carries the ranking signal; changing the semantic weight or the intent range changes AUC by at most 0.001.
Retraining. We retrained SAO with each variant under the Table 8 protocol:
All differences are within about one point, so the results are robust to these choices. The deployed weights are slightly better on precision, S@1 and Bl@1, so we keep them as the default.
Versus SQL-BLEU: BLEU scores below 0.3 on 43% of correct (rating-2) completions, mostly short correct prefixes penalized by its brevity term; the judge does so on 7%.
Q2/Q3: Rating 1 vs. 2, and a correct opening followed by a minor error. A correct opening followed by a grounded, non-contradictory deviation already gets rating 1 (partial credit). After a correct opening, an unsupported or contradictory continuation receives 0. At a normal cursor, rating 2 means everything emitted is correct for the user's intended edit; a short, correct prefix is enough. Example, with
'EMEA'and the columndsin the context: the user later executedWHERE region = 'EMEA' AND ds >= …, and the cursor is afterWHERE region:= 'EMEA'→ 2= 'EMEA' AND ds IS NOT NULL→ 1= 'APAC'(not in context) → 0Real users follow this split: acceptance is 9% / 15% / 52% for ratings 0 / 1 / 2 (Table 9). Rating-0 completions are rarely accepted, while short, fully correct ones are accepted most often, so pass@1 counts only rating 2, while mean r gives rating 1 partial credit. The examples are now in App. C.
W1: Other dialects. Porting mainly requires three changes: the SQL parser dialect used by the grounding gate, the list of built-ins treated as universal constants, and the prompts. The reconstruction, trimming and gate-first reward stay the same. Evaluating another dialect needs editing logs from it, which we leave to future work. App. A now describes this.