- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how the location of business definitions affects text-to-SQL agents. It assigns a definition to each ambiguous question and provides it through in places, including the prompt, a database glossary, and a pre-executed lookup. Experiments on tasks constructed over BIRD and KaggleDBQA databases show that the tested agents often ignore stored definitions, while instructions and direct delivery can substantially improve accuracy.
Critical strengths
The paper studies a meaningful question, since business knowledge can be spread across different sources. However, the problem modeling is too simple.
The experiments include useful controls. The main findings also hold on ten new concepts and across several model families, which makes the results more convincing.
The main idea is easy to understand, and the concrete examples help explain the different experimental settings.
Critical weaknesses
-
This paper provides limited novelty and insight. The paper mainly relocates the same knowledge definition and measures changes in model performance. Related work on ambiguity in text-to-SQL, such as BIRD-Interact, already considers knowledge accessed through user clarification, knowledge bases, and metadata files. Although the paper identifies “definition relocation” as its distinguishing feature, I find little additional insight beyond showing that glossary definitions are harder under the authors-defined agent harness (only two tools), while a simple instruction changes this behavior for some models.
-
The scope the paper considers is very small. In practice, relevant knowledge may be distributed across database contents, glossaries, semantic layers, historical SQL, user input, and unstructured documents. Multiple sources may even contain conflicting definitions that require conflict resolution via cross-checking and asking user. The paper’s controlled placements and simple prompt-versus-glossary conflict provide limited evidence about these more complex cases, which weakens the practical significance of its findings.
-
The prompts and presentation are unclear. The complete baseline prompts, particularly for the Stored condition, are not shown in the paper. The opening claim, “Text-to-SQL benchmarks give agents two kinds of missing information at once,” is not well supported, and its relevance to real-world knowledge organization is not well established. Clearer connections to more professional terminologies such as business semantics, ontologies, and data dictionaries would help explain the framing. Additionally, an overview figure would also make the paper easier to follow.
Questions for authors
See the above section content.
3 comments
Thanks for the clarification. I’m raising Informativeness from 1 to 2. However, my overall rating remains Weak Reject since the reserach question is a little bit small and the insights I can get is limited. The authour should also improve the writing skills.
Thank you for raising Informativeness. On the size of the question and the insights, here is what a builder of a data agent learns from the paper that they could not learn before:
- A stored glossary is worth nothing to most open-weight agents. They query it in 0 to 2% of sessions, so a stored definition adds points . The same rows served by the system add .
- Telling the agent to look works only for some models. One sentence lifts GPT-5 from 45.9% to 93.3%. Mistral queries in 76% of sessions and gains only 2.9 points, and Qwen3-8B and Qwen3-14B do not improve.
- The same instruction can override the user. With a conflicting glossary, "check the database for business terms" lowers Claude Opus 5.5's agreement with the user's stated definition by 44.8 points. One clause giving the user precedence restores it to 98.5%.
- The query rate predicts what a stored definition is worth, within 2.9 points on average over the seven cells where agents query.
Our study isolates one step that every data agent with a glossary, semantic layer, or data dictionary goes through. Each of the four results above is a deployment decision: either where to put definitions or which instruction to add.
On writing, the revision adds:
- an overview figure with the method and one worked item in every position (Figure 1)
- the complete prompts (Appendix L)
- standard terms such as business glossary, data dictionary and semantic layer (Section 2)
- a softened opening claim in the abstract and introduction
If a specific section is unclear or a specific insight is missing, we will address it. Without such a pointer, we consider the three weaknesses in the review resolved.
Reply to Reviewer HdDG
The two-tool harness does not prevent the lookup: Claude Opus 5.5 queries the glossary in every session on that harness, without being told. Our answers follow your three points.
W1. The harness, and novelty. run_sql accepts any read-only statement, and open-weight settings stop after 1 to 2 statements of the 12 allowed. You are right that looking depends on the loop and the model, and the paper measures both. Three coding agents run the same items under their own loops and tools and gain 35.6 to 71.3 points from a stored definition (Table 4). On our harness, Claude Opus 5.5 looks in 100% of sessions and GPT-5.2 in 16%. On novelty, BIRD-Interact offers several knowledge sources but does not vary where a given fact lives, so it cannot say what a fact is worth in one place against another, or which place wins when two disagree. We measure both. A stored definition is worth +0.3 points [−1.5,+2.8] to open-weight agents, while the same rows served are worth +29.4 [+19.2,+41.0]. With a conflicting glossary, the instruction to look lowers the share of Claude Opus 5.5's answers that follow the definition stated in the prompt (a line labeled Hint) by 44.8 points [−55.5,−34.3] over 20 concepts, while GPT-5.2 barely moves (−4.2). A second run and a wording control confirm both, and the shift is 3.3 points when the definition is inside the question instead (Table 11).
W2. Scope. The paper covers five places a definition can live (prompt, glossary table, schema text, tool result, the data owner) and a conflict between two of them. For that conflict the revision also gives a remedy: one clause that gives the user's statement precedence restores Claude Opus 5.5 to 98.5% and 98.6% on the two sets, from 43.7% and 43.6% (Table 11). We agree that historical SQL and unstructured documents are not covered, and Section 8 now says so. The single-definition case is the necessary first measurement: an agent that does not retrieve a definition from one glossary table is unlikely to cross-check several sources, and five of our ten settings do not retrieve it.
W3. Prompts, the opening claim, terminology, overview figure. You are right that the paper omitted the prompts. Appendix L now gives them in full. The system prompt is four sentences: "You are a data analyst answering a user's question with a SQLite query. You may run read-only SQL with run_sql to explore the database before answering. When done, call submit_answer with the final query. You have at most 12 tool calls." In the Stored position the prompt is unchanged except that the schema includes the CREATE TABLE statement of glossary(term, meaning), with no rows shown. Figure 1 is the overview, with the method in five steps and one item in every position. On the opening claim, BIRD supplies one evidence string per question and KaggleDBQA one documentation file per database, so every kind of missing information arrives through the same channel at once. The abstract now says "through one channel" instead of "at once", and the introduction says "in the same hint" instead of "in the same sentence". Appendix J shows why we do not separate the kinds by labeling those hints: annotators agree at κ between 0.016 and 0.399. On terminology, our glossary table is a business glossary or data dictionary entry, and the Delivered position is a semantic-layer document in context. Section 2 now uses these terms.