acceptodds
Under review as a conference paper at ICLR 2027

The Truth Is Out There: Separating Owner Definitions from Database Facts in Text-to-SQL

Abstract

Text-to-SQL benchmarks give agents two kinds of missing information at once. The database settles some of it, such as which column holds a concept or how its values are encoded. The data's owner decided the rest, such as where a threshold lies or which categories count, and no query recovers it. Because benchmark hints supply both together, a score cannot say which one an agent lacked. We separate them with definition relocation. For an underspecified question, we write several definitions an analyst could hold, make one of them correct, and move it between the prompt, a glossary table in the database, the schema text, the agent's transcript, and the data owner. Each definition implies a different answer, so the answer shows which one the agent acted on. Where the definition sits matters as much as whether it exists. Open-weight models almost never query the glossary, so a definition stored there is worth nothing to them, while the same rows returned by a query run on their behalf close 35 to 85% of the gap to stating the definition in the prompt. Agents that explore on their own do recover stored definitions, one sentence of instruction makes a hosted model do the same, and how often an agent queries the glossary predicts what a stored definition is worth to it. Documentation of what the database already settles behaves differently: it helps small models and adds nothing measurable for strong ones. A second set of concepts reproduces these results.

Primary area:
datasets and benchmarks
Keywords:
Text-to-SQL, LLM agents, tool-calling agents, database exploration, ambiguity resolution, business definitions, glossary retrieval, owner definitions, knowledge conflicts, semantic layers, BIRD benchmark, KaggleDBQA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HdDGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the location of business definitions affects text-to-SQL agents. It assigns a definition to each ambiguous question and provides it through in places, including the prompt, a database glossary, and a pre-executed lookup. Experiments on tasks constructed over BIRD and KaggleDBQA databases show that the tested agents often ignore stored definitions, while instructions and direct delivery can substantially improve accuracy.

Critical strengths

The paper studies a meaningful question, since business knowledge can be spread across different sources. However, the problem modeling is too simple.

The experiments include useful controls. The main findings also hold on ten new concepts and across several model families, which makes the results more convincing.

The main idea is easy to understand, and the concrete examples help explain the different experimental settings.

Critical weaknesses
  1. This paper provides limited novelty and insight. The paper mainly relocates the same knowledge definition and measures changes in model performance. Related work on ambiguity in text-to-SQL, such as BIRD-Interact, already considers knowledge accessed through user clarification, knowledge bases, and metadata files. Although the paper identifies “definition relocation” as its distinguishing feature, I find little additional insight beyond showing that glossary definitions are harder under the authors-defined agent harness (only two tools), while a simple instruction changes this behavior for some models.

  2. The scope the paper considers is very small. In practice, relevant knowledge may be distributed across database contents, glossaries, semantic layers, historical SQL, user input, and unstructured documents. Multiple sources may even contain conflicting definitions that require conflict resolution via cross-checking and asking user. The paper’s controlled placements and simple prompt-versus-glossary conflict provide limited evidence about these more complex cases, which weakens the practical significance of its findings.

  3. The prompts and presentation are unclear. The complete baseline prompts, particularly for the Stored condition, are not shown in the paper. The opening claim, “Text-to-SQL benchmarks give agents two kinds of missing information at once,” is not well supported, and its relevance to real-world knowledge organization is not well established. Clearer connections to more professional terminologies such as business semantics, ontologies, and data dictionaries would help explain the framing. Additionally, an overview figure would also make the paper easier to follow.

Questions for authors

See the above section content.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer HdDG

The two-tool harness does not prevent the lookup: Claude Opus 5.5 queries the glossary in every session on that harness, without being told. Our answers follow your three points.

W1. The harness, and novelty. run_sql accepts any read-only statement, and open-weight settings stop after 1 to 2 statements of the 12 allowed. You are right that looking depends on the loop and the model, and the paper measures both. Three coding agents run the same items under their own loops and tools and gain 35.6 to 71.3 points from a stored definition (Table 4). On our harness, Claude Opus 5.5 looks in 100% of sessions and GPT-5.2 in 16%. On novelty, BIRD-Interact offers several knowledge sources but does not vary where a given fact lives, so it cannot say what a fact is worth in one place against another, or which place wins when two disagree. We measure both. A stored definition is worth +0.3+0.3 points [−1.5,+2.8][-1.5, +2.8] to open-weight agents, while the same rows served are worth +29.4+29.4 [+19.2,+41.0][+19.2, +41.0]. With a conflicting glossary, the instruction to look lowers the share of Claude Opus 5.5's answers that follow the definition stated in the prompt (a line labeled Hint) by 44.8 points [−55.5,−34.3][-55.5, -34.3] over 20 concepts, while GPT-5.2 barely moves (−4.2-4.2). A second run and a wording control confirm both, and the shift is 3.3 points when the definition is inside the question instead (Table 11).

W2. Scope. The paper covers five places a definition can live (prompt, glossary table, schema text, tool result, the data owner) and a conflict between two of them. For that conflict the revision also gives a remedy: one clause that gives the user's statement precedence restores Claude Opus 5.5 to 98.5% and 98.6% on the two sets, from 43.7% and 43.6% (Table 11). We agree that historical SQL and unstructured documents are not covered, and Section 8 now says so. The single-definition case is the necessary first measurement: an agent that does not retrieve a definition from one glossary table is unlikely to cross-check several sources, and five of our ten settings do not retrieve it.

W3. Prompts, the opening claim, terminology, overview figure. You are right that the paper omitted the prompts. Appendix L now gives them in full. The system prompt is four sentences: "You are a data analyst answering a user's question with a SQLite query. You may run read-only SQL with run_sql to explore the database before answering. When done, call submit_answer with the final query. You have at most 12 tool calls." In the Stored position the prompt is unchanged except that the schema includes the CREATE TABLE statement of glossary(term, meaning), with no rows shown. Figure 1 is the overview, with the method in five steps and one item in every position. On the opening claim, BIRD supplies one evidence string per question and KaggleDBQA one documentation file per database, so every kind of missing information arrives through the same channel at once. The abstract now says "through one channel" instead of "at once", and the introduction says "in the same hint" instead of "in the same sentence". Appendix J shows why we do not separate the kinds by labeling those hints: annotators agree at κ\kappa between 0.016 and 0.399. On terminology, our glossary table is a business glossary or data dictionary entry, and the Delivered position is a semantic-layer document in context. Section 2 now uses these terms.

Thanks for the clarification. I’m raising Informativeness from 1 to 2. However, my overall rating remains Weak Reject since the reserach question is a little bit small and the insights I can get is limited. The authour should also improve the writing skills.

Thank you for raising Informativeness. On the size of the question and the insights, here is what a builder of a data agent learns from the paper that they could not learn before:

  1. A stored glossary is worth nothing to most open-weight agents. They query it in 0 to 2% of sessions, so a stored definition adds +0.3+0.3 points [−1.5,+2.8][-1.5, +2.8]. The same rows served by the system add +29.4+29.4 [+19.2,+41.0][+19.2, +41.0].
  2. Telling the agent to look works only for some models. One sentence lifts GPT-5 from 45.9% to 93.3%. Mistral queries in 76% of sessions and gains only 2.9 points, and Qwen3-8B and Qwen3-14B do not improve.
  3. The same instruction can override the user. With a conflicting glossary, "check the database for business terms" lowers Claude Opus 5.5's agreement with the user's stated definition by 44.8 points. One clause giving the user precedence restores it to 98.5%.
  4. The query rate predicts what a stored definition is worth, within 2.9 points on average over the seven cells where agents query.

Our study isolates one step that every data agent with a glossary, semantic layer, or data dictionary goes through. Each of the four results above is a deployment decision: either where to put definitions or which instruction to add.

On writing, the revision adds:

  • an overview figure with the method and one worked item in every position (Figure 1)
  • the complete prompts (Appendix L)
  • standard terms such as business glossary, data dictionary and semantic layer (Section 2)
  • a softened opening claim in the abstract and introduction

If a specific section is unclear or a specific insight is missing, we will address it. Without such a pointer, we consider the three weaknesses in the review resolved.

Reviewer_fup3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the availability, retrieval, and use of business definitions in Text-to-SQL. It introduces definition relocation: assigning a definition to an underspecified question and placing the same definition in the user prompt, a database glossary, schema text, or a pre-injected tool response. The authors construct RD-BIRD and RD-Kaggle and compare models’ query behavior and answers across these conditions. Under the tested harness, several open-weight models rarely query the glossary proactively. Directly supplying definitions substantially improves performance, although weaker models still struggle to use them effectively.

Critical strengths
  • The problem is practically relevant. Distinguishing missing business definitions, information acquisition failures, and SQL generation difficulties helps identify the sources of Text-to-SQL errors more accurately.

  • The experimental design is interpretable. Candidate definitions yield different answers, and the same definition is supplied under different conditions, allowing the evaluation to test whether model outputs follow the assigned business convention.

Critical weaknesses
  • Model coverage in the core experiments remains limited. The OpenAI models in Tables 1 and 7 are GPT-5 and GPT-5 mini. Although Appendix Table 9 includes newer models, they are not evaluated under the core definition-relocation conditions. The robustness of the main behavioral findings across model generations therefore remains unclear.

  • Concept coverage is limited, and applicability to realistic business settings is not sufficiently established. The main RD-Kaggle experiment contains only 10 concepts and 50 questions. RD-BIRD and the additional concept set provide useful validation, but the latter still uses KaggleDBQA databases and the same construction procedure. The findings have limited coverage of more complex cases, such as rules distributed across documents, combinations of definitions, and user-dependent conventions.

  • The analysis of retrieval and use failures remains coarse-grained. Querying the glossary does not necessarily retrieve the relevant definition. An incorrect answer after receiving it may result from definition interpretation, schema linking, or SQL construction errors. Appendix C discusses some retrieval shortcomings, but systematic stage-level statistics and error categorization are still needed to explain the main performance gaps.

  • The additional diagnostic value over prior work needs clarification. Tk-Boost already ablates retrieval and application, while BIRD-Interact’s Memory Grafting examines differences between information acquisition and SQL generation capabilities. Assigning definitions and relocating them across channels offers some novelty, but the paper should more concretely explain which failure modes this design distinguishes that existing diagnostics cannot.

  • The main text lacks a complete example illustrating the experimental design. The wildfire example is helpful, but it does not show the inputs, tool responses, and expected answers across conditions. The only figure and the key Served results appear in Appendix H and Table 7, respectively. A worked example, a method diagram, and a consolidated presentation of the core comparisons would improve clarity.

Questions for authors
  • Could the authors evaluate some of the newer models under the core conditions and report glossary query rates and the Stored–Absent and Served–Stored accuracy differences?

  • Could the authors further explain the concept selection criteria and provide a small-scale replication on independent databases or externally sourced business questions? Do the findings hold for tasks requiring multiple definitions to be combined?

  • Could the authors separately report the rates of attempting a glossary query, successfully obtaining the relevant definition, and producing a correct final answer after obtaining it, together with a categorization of failures?

  • What additional issues can it identify beyond Tk-Boost’s retrieval/application ablations and BIRD-Interact’s Memory Grafting? Why does Table 2 classify BIRD-Interact as having no retrieval intervention?

  • Please show a question’s candidate definitions and corresponding answers, along with the exact information supplied to the model under Stored, Delivered, Served, and Stated.

  • As a potential extension, could this design apply to NL-to-PL/SQL? What changes to definition relocation and outcome evaluation would be needed when generating PL/SQL programs involving control flow, state updates, or transactions from natural language?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer fup3

We ran two newer models, GPT-5.2 and Claude Opus 5.5, under the core conditions on 20 concepts. GPT-5.2 reproduces the retrieval contrasts, Claude Opus 5.5 retrieves without being told, and the two differ when the glossary conflicts with the prompt. The stage-level statistics and the worked example you requested are added below.

Q1 (W1). Newer models under the core conditions. Both ran on our harness on both KaggleDBQA sets (Table 10, every position except Delivered). GPT-5.2 queries the glossary in 16% and 32% of Stored sessions. Its Stored minus Absent is +14.7+14.7 [+2.7,+28.5][+2.7, +28.5] pooled (+6.3+6.3 [−4.0,+18.3][-4.0, +18.3] on the first set alone) and its Served minus Stored is +48.5+48.5 [+33.8,+62.7][+33.8, +62.7]. Claude Opus 5.5 queries in 100% of Stored sessions. Its Stored minus Absent is +72.8+72.8 [+63.3,+82.2][+63.3, +82.2] and its Served minus Stored is −0.3-0.3 [−2.0,+1.0][-2.0, +1.0], so retrieval no longer limits it. Under the instruction its agreement with the prompt's Hint line falls by 44.8 points [−55.5,−34.3][-55.5, -34.3], against 4.2 for GPT-5.2. Both reproduce in a second run, and for Claude Opus 5.5 the fall is 3.3 points when the definition is inside the question instead (Table 11). The submission already had GLM 5.3 in all six positions (Tables 1 and 7: 41% query rate, +23.7+23.7 from storage, +37.0+37.0 more from serving), and Claude Opus 5 and Codex in Absent and Stored under their own loops (Table 4: +54.1+54.1 and +45.9+45.9 on RD-Kaggle). The other Table 9 models were not run in these conditions.

Q2 (W2). Concept coverage and independent replication. We wrote each concept as a business term the question uses without defining, with 2 to 3 definitions an analyst could hold, each giving a different answer on the shipped data (checked by executing its reference query) and none inferable from the value distribution. To guard against favorable choices, a coding agent drafted the second set. RD-BIRD replicates on different databases (9 concepts, a different definition format) and reproduces the stored contrast (Table 3). The second KaggleDBQA set has 10 new concepts and reproduces both contrasts, now also for the two new models (Appendices D and M). Pooled over the 20 KaggleDBQA concepts, for the open-weight settings Stored minus Absent is +0.3+0.3 [−1.5,+2.8][-1.5, +2.8] and Served minus Stored is +29.4+29.4 [+19.2,+41.0][+19.2, +41.0]. Combined definitions and rules spread over documents are outside the design, as Section 8 now states. One definition in one table is the easiest version of the problem, and open-weight agents already fail it.

Q3 (W3). Stage-level statistics and failure categories (Table 12). "Retrieved" means a glossary statement returned the row with the assigned definition.

Setting (135 sessions each)QueriedRetrievedCorrect after retrieving
Qwen3-32B2110 of 1
GPT-5 mini111111 of 11
GPT-5333332 of 33
GLM 5.3555553 of 55
GPT-5.2211715 of 17
Claude Opus 5.5135135134 of 135
Mistral + instruction1036515 of 65
GPT-5 + instruction131128125 of 128

Qwen3-32B uses the glossary only as a filter inside its answer query, so it retrieves in 1 of 21 sessions. After certain receipt (Served, Table 14), open-weight settings submit a query that implements none of the candidate definitions in 65 to 93 of 135 sessions, which points to schema linking or SQL construction, and a query that implements a different candidate in 9 to 19. For GPT-5 the counts are 1 and 4.

Q4 (W4). What the design identifies beyond Tk-Boost and memory grafting. It identifies which source an agent follows when two locations disagree (Q1), and it separates the three failures in the table above (no query, a query that misses the definition, a retrieved definition that is not applied), each measured against the same fixed definition. Neither prior design can do either, because neither varies the location of a fact. You are right about Table 2: it now credits memory grafting as an intervention on acquisition.

Q5 (W5). Method diagram and example: Figure 1. Question: "How many large wildfires were there in California in 2007?" Candidates: at least 300 acres (97), 1,000 (47), 5,000 (18); assigned 1,000, so the expected answer is 47. The system prompt and question are identical in every position.

  • Absent: schema and question.
  • Stated: the same, then Hint: A large wildfire is a fire that burned at least 1,000 acres.
  • Stored: schema and question. The schema lists glossary(term, meaning) but not its five rows, one of them the assigned definition.
  • Delivered: the same five rows printed in the schema.
  • Served: as Stored, with the transcript opening on a run_sql call, SELECT * FROM glossary, and its five rows.
  • Conflicting: the Stated prompt, with a glossary that holds a different candidate.

Q6. PL/SQL. Relocation carries over unchanged, since it concerns where the definition lives. Scoring would compare final database states instead of result sets.

I thank the authors for the additional experiments and clarifications. However, my concerns regarding W2–W4 and Q6 remain only partially resolved, so I maintain my original score.

Thank you for engaging with the new results. Since W2 to W4 and Q6 are described as partially resolved, we list below in more detail how the revised paper addresses each of them.

W2 (concept coverage and realism). The findings hold on 29 concepts in three sets: 9 on BIRD databases, 10 written by us on KaggleDBQA, and 10 drafted by a coding agent so that our own choices could not favor the result. Each concept is a business term the question uses without defining, with 2 to 3 definitions that give different answers on the data, checked by executing each reference query. Pooled over the 20 KaggleDBQA concepts, a stored definition is worth +0.3+0.3 points [−1.5,+2.8][-1.5, +2.8] to open-weight agents and serving it is worth +29.4+29.4 [+19.2,+41.0][+19.2, +41.0] more. GPT-5.2 reproduces both contrasts, and Claude Opus 5.5 retrieves without being told.

On the complex cases you list:

  • User-dependent conventions are the core of the design. Each question is crossed with every candidate definition, so identical wording has a different correct answer depending on which owner's convention is assigned. The agent can succeed only by acquiring that convention. The Conflicting position adds the case where the user's convention and the database's disagree. What we do not test is inferring a convention from who is asking when it is never stated, and Section 8 says so.
  • Rules spread across documents and combined definitions are outside the design, as Section 8 states. One definition in one table is the easiest version of the problem, and open-weight agents already fail it (0 to 2% query rate, apart from Qwen3-32B at 16%), so harder versions are unlikely to reverse this finding.

W3 (stage-level failures). Table 12 reports each stage you asked for: attempted a glossary query, retrieved the assigned definition, and answered correctly after retrieving it. Table 14 then classifies every answer once the definition has certainly been received (Served rows, 135 sessions each):

SettingAssigned definitionDifferent candidate (misapplied definition)No candidate (schema linking or SQL)
Qwen3-4B33993
Qwen3-8B421875
Qwen3-14B421479
Qwen3-32B541665
Mistral-Small-24B371979
GPT-5 mini12087
GPT-513041

For open-weight settings, most failures after receipt are queries that implement no candidate, which points to schema linking or SQL construction. Before receipt, Table 12 shows where each setting fails. Qwen3-32B queries in 21 sessions and retrieves the definition in 1, because it uses the glossary as a filter inside its answer query. Mistral with the instruction queries in 103 of 135 sessions, retrieves the definition in 65, and answers correctly in 15 of those 65.

W4 (diagnostic value over prior work). Relocation measures three things the other designs cannot:

  1. The decision to look. Tk-Boost's pipeline retrieves on the agent's behalf, and memory grafting hands GPT-5 another model's dialogue, so in both the agent never chooses to look. Our stored position shows that open-weight agents query the glossary in 0 to 2% of sessions, apart from Qwen3-32B at 16%.
  2. Lucky defaults. Neither Tk-Boost nor BIRD-Interact assigns the correct meaning independently of the data, so a default that happens to match looks like retrieval. Without a definition, GPT-5 gives the same answer to every copy of 64% of questions.
  3. Which source wins when two disagree. An instruction to check the glossary lowers Claude Opus 5.5's agreement with the user's stated definition by 44.8 points, against 4.2 for GPT-5.2. One precedence clause restores it to 98.5%.

Q6 (PL/SQL). Relocation carries over unchanged, since it concerns where the definition lives, not what the agent writes. Scoring would change as follows:

  • Each candidate definition gets a reference program.
  • Each program runs in a transaction from the same database snapshot, its final state is recorded, and the transaction is rolled back.
  • The agent's program is scored by which candidate's final state it reproduces.

Control flow and updates change how the answer is produced but not how it is compared, so lucky defaults remain identifiable. We have not run this extension and leave it to future work.

We are glad to answer any specific question that remains. Without one, we consider W2 to W4 and Q6 resolved.

Reviewer_axmeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies text-to-SQL questions whose answers depend on an unstated definition (e.g., the threshold for “large” in “How many large wildfires occurred?”). For each question, the authors designate one of several plausible definitions as correct and compare agent accuracy when it is omitted, stated in the question, stored in a database glossary, or provided through schema text, tool results, or scripted replies from the data owner. The tested Qwen and Mistral models rarely retrieve stored definitions without instructions, while directly supplying the information improves accuracy. These effects vary with the instructions, agent execution loop, and definition format. A separate experiment examines the benefit of existing database documentation across models, treating that documentation as information recoverable from the database.

Critical strengths

S1. The paper is well written and clearly organized.

S2. The experiments cover multiple models, agent workflows, instruction settings, and definition formats.

Critical weaknesses

W1. The significance of the central research question is insufficiently motivated.

It remains unclear why documenting agents’ failure to seek definitions and their sensitivity to definition placement addresses a sufficiently important research problem. A natural baseline is to require definition lookup and ask the user when the definition is unavailable. The instruction and automatic-delivery experiments already show acquisition gains and remaining application/format effects. However, the paper should explain what substantial insight these measurements provide beyond motivating an explicit lookup-and-clarify workflow.

W2. The methodological advance over prior work is insufficiently established.

Tk-Boost [1] varies knowledge retrieval and application, while BIRD-Interact [2] supplies GPT-5 with other models’ clarification histories to probe acquisition separately from SQL generation. This paper adds independent definition assignment, controlled relocation, and definition tracking. However, its comparison in Table 2 does not address BIRD-Interact’s memory-grafting intervention, and the diagnostic advantage over these existing interventions remains underexplained.

W3. The broad empirical takeaways offer limited novelty.

BIRD-Interact [2] already reports limited knowledge exploration and improved SQL generation after supplying clarification histories; Tk-Boost [1] shows that retrieval and application choices affect accuracy. The paper’s broad findings—rare glossary lookup, gains from supplied definitions, and workflow-dependent accuracy—largely reinforce this picture. Although the study adds finer measurements and model-specific observations, it should explain which conclusions extend beyond these established findings.

W4. The documentation and clarification comparisons need stronger controls.

First, the paper compares how much agents benefit from receiving an owner definition versus database documentation. It treats the documentation as information agents could discover by querying the database, but does not verify this assumption (§§6, 8). The difference in gains therefore does not establish that database-recoverable facts and owner definitions affect performance differently.

Second, the paper compares accuracy on questions where the agent asked the owner with accuracy across all questions when the definition is provided directly in the prompt (§5). Because the agent chooses when to ask, the two sets may differ in difficulty. Both conditions should be compared on the same questions before attributing any gap to the delivery method.

[1] Shubham Agarwal et al. Arming Data Agents with Tribal Knowledge. arXiv:2602.13521v1, 2026.

[2] Nan Huo et al. BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions. arXiv:2510.05318v2, 2025.

Questions for authors
  1. Why is measuring definition-placement effects important beyond motivating a lookup-and-clarify workflow?

  2. What additional diagnostic capability does your design provide over Tk-Boost’s ablations [1] and BIRD-Interact’s memory grafting [2]?

  3. Which main empirical conclusions extend beyond the findings already established in [1] and [2]?

  4. Can you verify that the documentation used in §6 contains information recoverable from the database?

  5. How does asking compare with direct provision when both are evaluated on the same subset of questions where the agent asked?

3 comments
AuthorsOpenReview ↗

Response to Reviewer axme

You were right that asked items are harder (Table 13); Section 5 now says so. We tested the lookup instruction you propose. It works for the GPT-5 models, fails on RD-Kaggle for the open-weight ones, and for Claude Opus 5.5 it moves 44.8 points of answers from a definition stated as a hint in the prompt to the glossary's.

Q1 (W1). Why measure placement, beyond motivating lookup-and-clarify? The measurements say which form of that workflow works and what it costs. Instructing the agent is unreliable. The lookup sentence raises Qwen3-8B's query rate to 4% and Qwen3-14B's to 7%, with no accuracy gain. Mistral queries in 76% of sessions and ends 3 points above its no-definition accuracy on RD-Kaggle (on RD-BIRD it rises from 16.7% to 44.9%). For the GPT-5 models the instruction works (45.9% to 93.3% for GPT-5). It also has a side effect that only relocation can show. When the glossary disagrees with the definition stated in the prompt (a line labeled Hint), GPT-5 follows the prompt in 60.0%, down from 91.9% (Table 1). Claude Opus 5.5 shows the effect more strongly (−44.8-44.8 points [−55.5,−34.3][-55.5, -34.3]), while GPT-5.2 reads the glossary and still follows the prompt (−4.2-4.2). With the definition inside the question instead, the shift is −3.3-3.3 for Claude Opus 5.5 and −14.7-14.7 for GPT-5 mini (Table 11). Clarification is rarely used even when it is free and always correct (6 to 35% of sessions, Table 8). Enforcing the lookup helps every setting that does not already retrieve, by 14.8 to 55.7 points over storing.

Q2 (W2). Diagnostic capability beyond Tk-Boost and memory grafting. First, what a failure to look costs. BIRD-Interact reports how agents spend their actions, but it does not move one fixed fact between locations, so it cannot say what the same fact would have been worth elsewhere. Tk-Boost's pipeline retrieves on the agent's behalf. Second, what one fixed fact is worth in each location. Memory grafting transfers another model's dialogue with a simulated user to one recipient (GPT-5), so the content varies with the donor, and no meaning is assigned among alternatives. Our served position holds the content fixed across ten settings. You are right that Table 2 did not credit memory grafting. It is corrected.

Q3 (W3). Conclusions that go beyond [1] and [2].

An instruction to consult stored definitions changes which source an agent follows, and by how much depends on the model and on how the prompt gives its definition. For four hosted models that read a conflicting glossary, the share of answers that follow it ranges from 5.2% (GPT-5.2) to 56.3% (Claude Opus 5.5). Both ends reproduce in a second run, and a neutral instruction to read every table produces no such shift (Table 11). Neither [1] nor [2] can measure this, because neither varies the location of one fixed fact. A stored definition is worth nothing to an agent that does not query, at +0.3+0.3 points [−1.5,+2.8][-1.5, +2.8] over 20 concepts and four settings, against +29.4+29.4 [+19.2,+41.0][+19.2, +41.0] when the same rows are served. The query rate predicts stored accuracy. A two-term mixture matches the observed cell within 2.9 points on average over the seven cells where more than 5% of sessions query (Table 5). It misses GPT-5.2 by 3.0 points (11.9 under the instruction) and Claude Opus 5.5 by 0.7 and 0.0. Form matters apart from retrieval. For open-weight settings the same definition is worth 6.4 to 17.7 points more as a sentence than as table rows (three of four intervals exclude zero).

Q4 (W4a). Is the documentation recoverable from the database? We now test it on behavior (Section 6, Appendix M). Take the 42 questions where the documentation turns a wrong Qwen3-4B answer into a right one. GPT-5 answers 27 of them without the documentation and 25 with it. Over the 100 questions where the documentation rescues any of the five open-weight settings, GPT-5 answers 59 without and 57 with. On the questions where the documentation matters to small models, GPT-5 gains nothing from it, while a stated definition is worth +60.7+60.7 points to the same model on RD-Kaggle. The check does not separate what GPT-5 recovers by querying from what it already holds, and the paper says so.

Q5 (W4b). Asking against stating on the same items. Table 13, scored as in Table 8 (the harness's own match, without the scorer's tolerance of Table 1):

SettingSessions that askedCorrect after askingCorrect on the same items when statedStated accuracy on the other items
Qwen3-4B130441%
Qwen3-8B263550%
Qwen3-32B31101147%
Mistral-Small-24B81058%
GPT-5 mini47404290%
GPT-534303292%

On the same items, asking and stating differ by at most 4 sessions. Open-weight settings ask on items they also fail when the definition is stated. Low accuracy after asking therefore reflects how these settings use a definition, and the ask rate is what limits the channel.

I maintain my recommendation of weak rejection. I consider W4 (Q4–Q5) addressed by the additional analysis and clarification. However, my concerns in W1–W3 (Q1–Q3) remain.

These concerns come down to one question: What does definition relocation tell us that existing analyses of information acquisition and use do not? W1 concerns the importance of this question, W2 the additional diagnostic value of the design, and W3 the significance of the resulting insights.

In detail:

  • W1/Q1: The additional experiments clarify when agents fail to retrieve or follow a definition. My concern is why measuring these differences across locations is an important research problem. The response provides more evidence of the differences, but does not sufficiently explain their significance.
  • W2/Q2: Fixing the definition and varying only its location gives a more controlled comparison. However, what does it reveal that existing acquisition/application interventions don’t? And why is it important?
  • W3/Q3: The source-conflict experiments give a more specific finding. But the result depends heavily on whether the definition is in the Hint or the question, and an explicit precedence rule fixes it. How is this more than the known instruction- and context-sensitivity of information use?

Relocation tells us what one fact is worth in each place an agent could find it, and which place the agent follows when two disagree. Existing analyses vary whether knowledge is supplied or how it is applied, so they cannot report either.

W1/Q1. Location decides whether a definition is worth anything, and what it costs.

  • A deployed system keeps the definition in a glossary, in documentation, or with a person, and scores do not transfer between these locations.
    • For the open-weight settings, the same definition is worth +0.3+0.3 points stored and +29.4+29.4 more when served.
    • For GPT-5.2 it is worth +6.3+6.3 stored and +55.7+55.7 more when served.
    • The gap between two locations is as large as the value of the definition itself. A benchmark score obtained with hints therefore does not predict the accuracy of an agent that has to find the same fact.
  • Prior work measures whether knowledge exists. Removing BIRD's evidence costs GPT-4 20.0 points on test (Li et al., 2023), and removing it costs 8.4 to 20.9 points across systems (Yun & Lee, 2025). Benchmarks then state the definition next to the question, which is the most favorable place for it.
  • Location also sets the cost.
    • Serving adds a median of 463 tokens per session and recovers 14.8 to 55.7 points over storing.
    • Asking the owner is cheaper, but agents ask in only 6 to 35% of sessions (Section 5).
    • Interactive benchmarks report cost per task: BIRD-Interact reports 0.04 USD to 0.60 USD per task and 0.03 USD for its user simulator. They do not report what one fact is worth in each place. Relocation gives every location a value in the same units, which this cost comparison requires.

W2/Q2. Relocation observes the agent's decision to look, which other interventions make for it.

  • Tk-Boost writes its statements from the agent's failures against ground-truth SQL, and its pipeline retrieves them after a first attempt (Agarwal et al., 2026). Its usage ablation holds content fixed and varies delivery, but every arm is on the pipeline side. No condition leaves the definition in the database for the agent to fetch.
  • Memory grafting hands GPT-5 another model's dialogue (Huo et al., 2026). BIRD-Interact reports success rate and normalized reward, and no result there prices one fact across locations. Its action analysis notes that models favor submitting and asking over knowledge retrieval. BIRD-Interact observes the behavior, and we measure what it costs. Open-weight settings query the glossary in 0 to 2% of sessions, apart from Qwen3-32B at 16%, and the same rows are worth +0.3+0.3 points left in the table and +29.4+29.4 more when served.
  • What a builder learns is where to intervene. Enforcing the lookup adds 14.8 to 55.7 points over storing, while instructing the agent leaves the accuracy of Qwen3-8B and Qwen3-14B unchanged.
  • Lucky defaults. Assignment separates following a definition from a default that happens to match, which execution accuracy cannot do. Without a definition, GPT-5 gives one fixed answer to every copy of 64% of questions. Concurrent work (PlanPool, Li et al., 2026) finds 36.9 to 40.4% of correct answers on BIRD-Interact-derived tasks to be ungrounded.

W3/Q3. The sensitivity is known. Its trigger, size, and remedy for stored definitions were not.

  • What is already known. We do not claim to discover instruction or context sensitivity. Conflicts between context sources (Xu et al., 2024), adoption of supplied content (Wu et al., 2024; Xie et al., 2024) and role-level precedence (Wallace et al., 2024) are established. For text-to-SQL, BIRD attributes 17.6% of sampled errors to misread evidence.
  • What is not yet measured. Closer work names the conflict without measuring it for definitions. ABISS treats conflicts within one knowledge base, AmbiSQL scores only their detection, and KC-Bench tests user-versus-database facts in retail agents. The instruction hierarchy orders roles, but a glossary row is data, so it does not decide this case.
  • Three properties make ours specific:
    1. The instruction, "check the database for business terms", is one a builder would add without a second thought.
    2. Reading the glossary does not cause the shift. Under a neutral instruction, the share changes by −0.5-0.5, +1.8+1.8 and −4.3-4.3 points.
    3. Under the same sentence the shift ranges from −4.2-4.2 (GPT-5.2) to −44.8-44.8 (Claude Opus 5.5).
  • The dependence on the Hint label is part of the finding, since a labeled hint is how benchmark evidence reaches the agent.
  • The precedence clause is the practical result. Without it, the instruction costs 34.2 to 44.8 points when the glossary disagrees with the prompt, a cost that had not been measured.

These results cover 20 concepts and one glossary table, and the paper states this scope. We consider W1 to W3 answered by the points above, and we ask you to reconsider your rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Our General Response for shared concerns raised throughout all reviews

Moving one assigned definition between locations yields a result that neither BIRD-Interact nor Tk-Boost can produce, shown here on 20 concepts. In our conflicting position the prompt states one definition, as a line labeled Hint, and a stored glossary holds another. An instruction to consult the glossary lowers the share of Claude Opus 5.5's answers that follow the prompt by 44.8 points [−55.5,−34.3][-55.5, -34.3] (95% interval, concept bootstrap), and by 52.7 in a second run. GPT-5 and GPT-5 mini shift by 34.2 and 37.7 points. GPT-5.2 reads the same glossary and still follows the prompt (−4.2-4.2 and +2.0+2.0 in two runs). We score against the prompt, so under that scoring these shifts are losses of accuracy. Four controls locate the cause (Table 11). A paraphrase of the instruction reproduces the loss (−41.8-41.8, −49.7-49.7 and −38.8-38.8 points for Claude Opus 5.5, GPT-5 and GPT-5 mini). A neutral sentence that only makes the agent read every table does not (at most −4.3-4.3). One added clause, "If the user states a definition, the user's definition takes precedence," removes it (+10.8+10.8, +1.3+1.3 and −3.0-3.0 against no instruction). With the same definition inside the question instead of on a Hint line, the shift falls to −3.3-3.3 [−6.7,−0.7][-6.7, -0.7] for Claude Opus 5.5 and −14.7-14.7 [−22.7,−7.0][-22.7, -7.0] for GPT-5 mini (GPT-5 was not run). Most of the shift therefore depends on the definition arriving as a labeled hint. An instruction to consult stored definitions therefore needs a stated precedence rule, and this can only be measured when the same fact sits in two places. Every change is highlighted in blue, and the new results sit in Appendix M.

  1. What the design shows that BIRD-Interact and Tk-Boost cannot. It shows what one fixed definition is worth in each location, how much of that value depends on the agent choosing to look, and which source the agent follows when two locations disagree. Neither varies the location of one fixed fact as the experimental variable.
Intended meaning assigned among alternativesSame fact moved between locationsAgent must decide to lookRetrieval intervened
BIRD-Interact, memory graftingnonoyesyes, for one model (GPT-5), with another model's dialogue
Tk-Boost ablationsnonono, the pipeline retrievesyes
This paperyesyes, six positionsyesyes, ten settings, with the content held fixed

Our served position is the analogue of memory grafting with the content controlled: one fixed definition and the same rows for every setting, with controls for tool demonstration and for form (Appendix E). Table 2 had two errors, and we corrected both. Memory grafting intervenes on acquisition, as Reviewers axme and fup3 note. Tk-Boost's pipeline retrieves for the agent, so its "must retrieve" entry is now empty.

  1. Both steps of "require lookup, then ask" are tested. The instruction to look works for GPT-5, GPT-5 mini and GPT-5.2, and it fails on RD-Kaggle for the three open-weight settings we tested. Agents rarely ask, even when the owner always answers correctly (6 to 35% of sessions, Table 8). A lookup that the system runs itself (our served position) gains 14.8 to 55.7 points over storing for every setting except Claude Opus 5.5, which already retrieves. The paper therefore supports the workflow when the system enforces it, and it shows that instructing the agent is unreliable and can change which source the agent follows.

  2. Breadth. The stored contrast holds on 29 concepts in three sets (9 on BIRD, 10 and 10 on KaggleDBQA), and the served contrast holds on the 20 KaggleDBQA concepts. For open-weight settings a stored definition is worth +0.3+0.3 points [−1.5,+2.8][-1.5, +2.8] and serving it is worth +29.4+29.4 [+19.2,+41.0][+19.2, +41.0]. Three newer models run on the shared harness: GLM 5.3 in all six positions, and GPT-5.2 and Claude Opus 5.5 on both KaggleDBQA sets (Table 10). GPT-5.2 mostly does not look: it gains +14.7+14.7 [+2.7,+28.5][+2.7, +28.5] from storage (+6.3+6.3 [−4.0,+18.3][-4.0, +18.3] on the first set alone) and +48.5+48.5 [+33.8,+62.7][+33.8, +62.7] more from serving. Opus 5.5 queries in every session, gains +72.8+72.8 from storage, and gains nothing further from serving.

New analyses, from the released records. Table 12 gives the stages of the stored position (fup3). Table 13 compares asking with stating on the same items (axme).

Changes in the revised PDF. (i) Short additions to the abstract, introduction, contributions, and conclusion on the source-conflict result. (ii) Figure 1: the method and one item in every position (the old Figure 1 is Figure 2). (iii) One paragraph in Section 4 on GPT-5.2 and Claude Opus 5.5. (iv) Appendix M: Tables 10 to 14 and the paired check of the documentation assumption. (v) Table 2 and the related-work text corrected. (vi) Appendix L with the complete prompts. (vii) Standard terms in Section 2. (viii) Small numeric corrections from our own audit of the release, none of which changes a sign or a conclusion. The main text is 10 pages.

AuthorsOpenReview ↗

Closing Statement

Every weakness in the three reviews has been answered, and no reviewer disputes a result in the paper. The ratings now rest on a single judgment, namely that the question is small. The reviews themselves do not support this judgment.

No soundness concern remains. All three reviewers rate Soundness as minor or fixable, and none raised a soundness point after our replies. Reviewer axme considers W4 addressed, and Reviewer HdDG raised Informativeness from 1 to 2.

The reviewers themselves say that the problem matters. Reviewer HdDG writes that "The paper studies a meaningful question, since business knowledge can be spread across different sources." Reviewer fup3 writes that "The problem is practically relevant." Reviewer HdDG also names the case that matters most in practice, "Multiple sources may even contain conflicting definitions", and this is the case that we measure. Reviewer axme calls an instruction to look up definitions "A natural baseline". We tested it. It fails on RD-Kaggle for the three open-weight settings that we tested, and for Claude Opus 5.5 it lowers agreement with the definition stated in the prompt by 44.8 points when the glossary disagrees (Table 11).

The remaining objections point to nothing specific. Reviewer fup3 writes that four points are "only partially resolved" and names no part that remains. Reviewer HdDG finds the question small and cites no work that answers it. Reviewer axme asks how the conflict result differs from known instruction sensitivity. It differs in three measured facts that no work cited in the reviews reports. A routine instruction triggers it, its size ranges from 4.2 to 44.8 points across two current models, and one precedence clause removes it. We remain available to answer any specific question. Without one, we consider all of these points resolved.

The question is timely, and the benchmark literature does not address it. Text-to-SQL benchmarks supply the missing knowledge in one place, most often next to the question, and the large body of systems evaluated on them inherits this placement. A deployed agent has to find the same knowledge in a glossary, in documentation, or from a person. We show that its value does not transfer. For open-weight agents, the same definition is worth +0.3+0.3 points when stored and +29.4+29.4 more when the system serves it. A benchmark score obtained with hints therefore does not predict the accuracy of an agent that has to look.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.