acceptodds
Under review as a conference paper at ICLR 2027

How Database Interfaces Shape Agent Performance: An Empirical Study of SQL and Executable Semantic Layers

Abstract

Database agents are typically built to query relational databases through SQL, but emerging semantic layers offer a fundamentally different agent-facing abstraction. We ask how the choice of agent-facing database interface shapes agent performance and failure modes. We systematically compare direct SQL access with an executable semantic-graph interface on BIRD Mini-Dev and LiveSQLBench-Lite. The results reveal that no interface dominates across workloads. On BIRD, graph settings are stronger, with the advantage increasing from 0.5 percentage points on Simple questions to 4.9 percentage points on Challenging questions. On LiveSQLBench-Lite, SQL leads by about four percentage points under both metadata conditions. Workload and agent-behavior analyses suggest that interface choice redistributes rather than eliminates reasoning difficulty: semantic structure can simplify identifying the relevant data and relationships, while analytically intensive workloads place greater pressure on constructing and finalizing queries. These findings establish the database interface as a first-class design choice for database agents and show that effective interface design depends on how reasoning complexity is allocated between reusable structure and query construction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text-to-SQL, Database Agents, Semantic Graphs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_36x3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper compares a GPT-5 database agent using direct PostgreSQL and SQL with the same agent using a virtual semantic graph queried through Cypher and PuppyGraph. Descriptive metadata is varied separately within each interface. On a common-answerable set of 450 BIRD Mini-Dev and 173 LiveSQLBench-Lite questions, the graph setting leads by 1.0–2.6 points on BIRD, while SQL leads by 4.1–4.2 points on LiveSQLBench-Lite. The proposed explanation is that executable relationships help when navigation dominates, whereas constructing multi-stage analytical queries remains harder through the graph interface.

Critical strengths
  1. The study treats the executable database interface as a first-class design choice. It holds the underlying data and agent model fixed, varies descriptive metadata within both interfaces, and audits graph answerability on the retained evaluation sets.
  2. The paper goes beyond an overall win rate. Its workload taxonomy, stratified results, and query-finalization traces offer a coherent account of why the preferred interface might reverse between relationship-centric and analytically heavy tasks.
Critical weaknesses
  1. The smaller interface differences are difficult to assess from the reported aggregates. Each condition is run three times, but the paper omits per-run scores and paired Graph–SQL outcomes. This is particularly consequential for the 1.0- and 2.6-point BIRD advantages. Rule-based incorrect verdicts can also be overturned by an LLM judge; the same adjudication policy is applied to both interfaces, but overturn counts by condition and scores before adjudication are not reported. Without these, a reader cannot tell whether the direction and size of the smaller effects are stable or sensitive to evaluation decisions.
  2. The workload account is plausible but not yet demonstrated across the reported runs. The regime contributions are explicitly computed for an analyzed round, and their sums differ from the three-run aggregate interface gaps. The paper does not identify that round or show whether the claimed relationship-centric and analytical-heavy contributions persist across runs. Likewise, the 75 graph-agent failures to finalize a query on LiveSQLBench-Lite are suggestive, but their distribution across task regimes and contribution to the SQL advantage are not quantified. These observations support a hypothesis about this SQL/Cypher/PuppyGraph comparison, not yet a general conclusion about semantic interfaces.
Questions for authors
  1. From the existing runs, can you report per-run Graph–SQL differences and paired question-level outcomes, along with adjudication overturn counts and overall scores before and after adjudication for each condition? Does the interface preference hold in every run and under rule-based scoring alone?
  2. Which run underlies the task-regime contribution analysis, and do the same dominant contributions appear in the other runs? Among the 75 graph-agent nonfinalizations, how many occur in each task regime, and how much of the observed SQL advantage do they account for?
3 comments

Question 1: Per-run results, paired outcomes, and LLM-based adjudication

Response. We thank the reviewer for asking us to report run-level results and the effect of adjudication. We compare strict rule-based execution accuracy with accuracy after the paper's LLM-based adjudication. Strict result comparison can mark an answer incorrect when it contains the requested information in a different output shape. LLM-based adjudication applies the same conservative answer-equivalence policy to SQL and Graph; it does not excuse differences in filters, join paths, aggregation grain, denominators, or requested answer content.

The first table answers two parts of the reviewer's question: how each condition scores before and after LLM-based adjudication in each run, and how many strict false negatives are reclassified for each condition. BIRD values are the aligned run views in the clean-results manifest, including documented targeted replacements; LiveSQLBench-Lite values use the aligned 173-question valid-case runs. Scores are shown as correct / valid questions (accuracy). Reclassification totals count run-cases across three runs, not unique questions.

BenchmarkSettingRule R1Adjudicated R1Rule R2Adjudicated R2Rule R3Adjudicated R3Reclassified across 3 runs
BIRDSQL structural280/450 (62.2%)350/450 (77.8%)289/450 (64.2%)345/450 (76.7%)297/450 (66.0%)346/450 (76.9%)175
BIRDSQL descriptive280/450 (62.2%)350/450 (77.8%)289/450 (64.2%)344/450 (76.4%)298/450 (66.2%)352/450 (78.2%)179
BIRDGraph structural296/450 (65.8%)352/450 (78.2%)294/450 (65.3%)351/450 (78.0%)298/450 (66.2%)351/450 (78.0%)166
BIRDGraph descriptive300/450 (66.7%)363/450 (80.7%)302/450 (67.1%)358/450 (79.6%)302/450 (67.1%)360/450 (80.0%)177
LiveSQLBench-LiteSQL structural79/173 (45.7%)99/173 (57.2%)74/173 (42.8%)98/173 (56.6%)76/173 (43.9%)100/173 (57.8%)68
LiveSQLBench-LiteSQL descriptive72/173 (41.6%)98/173 (56.6%)76/173 (43.9%)99/173 (57.2%)73/173 (42.2%)96/173 (55.5%)72
LiveSQLBench-LiteGraph structural54/173 (31.2%)91/173 (52.6%)54/173 (31.2%)93/173 (53.8%)52/173 (30.1%)91/173 (52.6%)115
LiveSQLBench-LiteGraph descriptive51/173 (29.5%)93/173 (53.8%)50/173 (28.9%)90/173 (52.0%)51/173 (29.5%)89/173 (51.4%)120

This table shows substantial adjudication gains in every condition, with the largest gains for Graph on LiveSQLBench-Lite. These are not arbitrary score increases: the LLM-based adjudication corrects strict-match false negatives when the answer is semantically equivalent but represented differently. No rule-based correct case was overturned as incorrect in these aligned views.

The second table answers whether the Graph–SQL difference persists run by run and reports the paired question-level outcomes. Graph-only and SQL-only count questions answered correctly by just one interface in that paired run. Graph−SQL is their net difference expressed in percentage points.

BenchmarkContextRoundRule Graph-onlyRule SQL-onlyRule Graph−SQLAdjudicated Graph-onlyAdjudicated SQL-onlyAdjudicated Graph−SQL
BIRDStructural13115+3.6 pp1412+0.4 pp
BIRDStructural22217+1.1 pp1812+1.3 pp
BIRDStructural31514+0.2 pp138+1.1 pp
BIRDDescriptive13111+4.4 pp185+2.9 pp
BIRDDescriptive22512+2.9 pp173+3.1 pp
BIRDDescriptive31410+0.9 pp135+1.8 pp
LiveSQLBench-LiteStructural1631−14.5 pp917−4.6 pp
LiveSQLBench-LiteStructural2626−11.6 pp813−2.9 pp
LiveSQLBench-LiteStructural3428−13.9 pp413−5.2 pp
LiveSQLBench-LiteDescriptive1324−12.1 pp510−2.9 pp
LiveSQLBench-LiteDescriptive2430−15.0 pp615−5.2 pp
LiveSQLBench-LiteDescriptive3527−12.7 pp613−4.0 pp

The paired results show that the preference is stable across all three runs and under both scoring procedures: Graph leads on BIRD, while SQL leads on LiveSQLBench-Lite. Thus, the direction of the workload-dependent finding is not introduced by LLM-based adjudication. Adjudication primarily changes the estimated size of the gap, substantially narrowing the strict-score SQL advantage on LiveSQLBench-Lite.

Example of a rule-based false negative

In BIRD Graph-descriptive Round 1, the question asks for valid administrator email addresses for a school matching specified conditions. The strict scorer marks the answer incorrect because the gold and Graph outputs have different shapes, even though they contain the same two requested addresses.

Gold result

admemail1admemail2
[email protected][email protected]

Graph result

admin_email
[email protected]
[email protected]

LLM-based adjudication recognizes that the two requested email values and the filtering conditions match, so it marks this answer correct. The mismatch is only in presentation: two columns in one gold row versus one column with two Graph rows.

Question 2: Task-regime analysis and unfinished queries

Response. We thank the reviewer for pointing out that the paper did not identify the run used for its task-regime analysis or report how unfinished queries were distributed across regimes. The manuscript draft indicates that the reported regime analysis was intended to use Round 2. That table reflects a single-round analysis on the subset used at the time, which excluded unfinished cases and other cases under the then-used filtering procedure. We did not retain a complete item-level record of those exclusions, so we cannot reconstruct that exact subset.

For a more transparent comparison across runs, we now report the weighted Graph–SQL effects for all three rounds on the full audited valid set: 450 BIRD questions and 173 LiveSQLBench-Lite questions. We retain unfinished cases and count them as incorrect. Values are weighted contributions in percentage points; the three-round mean is the arithmetic mean of the round-level values.

BenchmarkTask regimeStructural R1R2R3MeanDescriptive R1R2R3Mean
BIRDLocal/simple+0.22+0.00+0.22+0.15+0.44+0.22+0.67+0.44
BIRDRelationship-centric+0.22+1.11+0.22+0.52+2.00+2.67+1.11+1.93
BIRDAnalytical-heavy+0.00+0.22+0.67+0.30+0.44+0.22+0.00+0.22
LiveSQLBench-LiteLocal/simple−1.73−1.16−0.58−1.16−0.58−0.58−0.58−0.58
LiveSQLBench-LiteRelationship-centric−1.16+0.00−1.16−0.77+0.00−0.58−0.58−0.39
LiveSQLBench-LiteAnalytical-heavy−1.73−1.73−3.47−2.31−2.31−4.05−2.89−3.08

The unfinished-query counts below are run-cases across the three rounds, not unique questions. The 75 LiveSQLBench-Lite Graph nonfinalizations are concentrated in the analytical-heavy regime. Among these cases, the paired SQL setting was adjudicated correct in 5 structural and 7 descriptive cases. These cases account for 5 of the 22 net SQL-favoring paired outcomes in structural runs and 7 of the 21 in descriptive runs; the remaining outcomes reflect the other paired questions.

LiveSQLBench-Lite regimeGraph structural unfinishedGraph descriptive unfinishedGraph totalSQL correct on paired Graph unfinished: structuralSQL correct on paired Graph unfinished: descriptive
Local/simple31400
Relationship-centric45900
Analytical-heavy29336257
Total36397557

The three-round results show a consistent workload-level pattern: relationship-centric regimes favor Graph on BIRD, while analytical-heavy tasks favor SQL on LiveSQLBench-Lite. We will clarify the single-round nature of the original regime analysis and add the three-round results and unfinished-query breakdown.

Thank you for the additional experiments and clarifications. They address some of my concerns, so I am raising my soundness rating to 3.

Reviewer_XkeZOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether the choice of agent-facing database interface, e.g., direct relational SQL versus an executable semantic graph layer (Cypher over PuppyGraph), affects LLM-based database agent performance. A controlled 2×2 factorial design crosses the interface type (SQL vs. graph) with the amount of descriptive metadata (structural vs. descriptive), evaluated on a common answerable population from BIRD Mini-Dev (450 questions) and LiveSQLBench-Lite (173 questions). A workload-dependent interface reversal, where the graph interface outperforms SQL on BIRD (up to +4.9 pp on Challenging questions), while SQL leads on LiveSQLBench-Lite (approximately −4 pp), is the main finding. Workload and agent-behaviour analyses attribute this reversal to a redistribution of reasoning difficulty. Elaborately, the paper shows that semantic structure simplifies data identification (relationship navigation), but analytically intensive workloads with CTEs, window functions, and multi-stage computation place greater pressure on query construction, where SQL agents have an advantage. Therefore, the paper positions the database interface as a first-class design dimension for database agents.

Critical strengths

A genuinely novel research question with a clean experimental design. The paper is, to my knowledge, the first to systematically compare executable database interfaces for LLM agents while controlling for confounding factors (descriptive metadata, answerable population, stochastic variation). The 2×2 factorial design is the right tool for this question, and the careful separation of interface expressibility from agent usability (via oracle audits and common answerable populations) sets a methodological standard for future interface comparison studies. The question itself, examining the database interface as a first-class design variable, not merely a given, is important and likely to influence how the community thinks about building database agents.

Secondly, rather than concluding that one interface is uniformly better, the paper shows that the graph interface advantages grow with relational-navigation complexity (BIRD Challenging: +4.9 pp with descriptive metadata) while SQL advantages grow with analytical-composition complexity (LiveSQLBench-Lite Moderate: −5.4 pp). This is directly actionable: it motivates workload-aware interface selection or hybrid interfaces that combine semantic navigation with SQL-level analytical composition. The finding that the interface choice redistributes rather than eliminates reasoning difficulty is a conceptually rich insight.

Critical weaknesses

The generality of the reversal finding is undercut by single-model, single-engine evaluation. The paper's central claim — that the interface effect is workload-dependent — rests on experiments with one LLM and one graph engine. Cypher proficiency varies substantially across LLMs: GPT-5 is a relatively strong SQL generator, and it is plausible that its Cypher generation capability is weaker simply due to training distribution. If the reversal disappears with a model that is equally proficient in SQL and Cypher, the finding would need to be reframed as a model-proficiency effect rather than an interface-workload effect. Similarly, PuppyGraph's virtual-graph mapping introduces its own translation layer with specific performance characteristics — the paper cannot distinguish whether the LiveSQLBench-Lite disadvantage is a property of graph interfaces in general or of this particular mapping implementation. Without at least one additional model and one additional graph engine, the reader must take the generality claim on faith.

W2: Modest effect sizes without statistical validation. The performance differences in several important cells of Table 2 and Table 3 are small (0.5–2.6 pp on BIRD, 0.8–5.4 pp on LiveSQLBench-Lite). Three runs per configuration is better than one, but the paper reports only aggregated accuracy without confidence intervals, variance estimates, or significance tests. At the +0.5 pp level (BIRD Simple, descriptive), the difference could easily fall within the run-to-run noise band. The paper would benefit from a paired bootstrap or McNemar's test at the question level, which would also enable the reader to distinguish reliably significant reversals from noisy estimates. Without this, some of the more fine-grained claims (e.g., "the advantage increases from 0.5 on Simple to 4.9 on Challenging") rest on unvalidated numerical comparisons.

Questions for authors

The paper is very well-written. The conceptual framework distinguishing interface expressibility from agent usability, and semantic structure from descriptive metadata, is both precise and helpful. The formal decision-chain notation (question--> objects --> program --> execution) makes the two interfaces immediately comparable. Table 1 (factorial settings) and Figure 1 (overview) are well-designed and economical. The consistent reporting of Δ (Graph − SQL) across all results tables makes the reversal pattern easy to track. The authors are careful to note that their workload analysis is correlational and that reference-SQL complexity serves as a proxy rather than a causal measure, avoiding overclaiming throughout.

The paper reframes a question I had not previously seen articulated in the database-agent literature, i.e., what should the interface between an agent and a database be? By separating this from the more familiar question of whether richer schema documentation helps text-to-SQL, the authors open a genuinely new design dimension. The workload-dependent reversal is non-obvious — a naive expectation might be that graph interfaces are uniformly better (because they abstract away physical schema details) or uniformly worse (because Cypher is less familiar to LLMs). The finding that graph interfaces help more as relational-navigation complexity increases but hurt when analytical composition dominates is a concise, actionable insight for practitioners building enterprise data agents. The decomposition into "relationship navigation" vs. "analytical composition" provides a useful conceptual vocabulary that extends beyond this specific paper.

The experimental methodology is commendable in the database-agent empirical literature. The common answerable population — with independent audits removing benchmark-invalid, non-comparable, and nondeterministic cases — is a strong safeguard against conflating interface limitations with agent failures. The factorial design cleanly separates the executable-structure effect from the descriptive-metadata effect. The oracle audit confirming that all 173 retained LiveSQLBench-Lite questions have graph-equivalent answers is essential for interpreting the SQL advantage on that benchmark as a genuine agent-usability difference rather than an interface-expressibility limitation. Three independent runs per configuration address stochastic variation. However, I have the following concerns:

(i) Single language model. All experiments use GPT-5. This is the most significant limitation. The workload-dependent reversal could be a property of how GPT-5 allocates reasoning effort between SQL and Cypher. A model that has seen substantially more Cypher training data, or one with weaker SQL capabilities, might show a different pattern. Even one additional model (e.g., Claude Opus 5 or Gemini) would substantially strengthen the claim that the reversal is a property of the interface–workload interaction rather than a model-specific artefact.

(ii) Single graph engine. PuppyGraph is the sole semantic-layer instantiation. Different graph engines (e.g., Neo4j, Amazon Neptune, or a SPARQL-based knowledge graph) have different Cypher dialects, analytical capabilities, and query-translation fidelities. The paper's conclusions about graph interfaces are entangled with PuppyGraph's specific mapping behaviour and Cypher dialect support.

(iii) Effect sizes and statistical reporting. The BIRD differences range from +0.5 to +4.9 pp and the LiveSQLBench-Lite differences from −0.8 to −5.4 pp. While three runs are reported, no confidence intervals or significance tests are provided. At these effect sizes, especially for the smaller deltas (+0.5, −0.8 pp), it is unclear whether the differences are statistically reliable. A simple bootstrap or paired test over question-level outcomes would resolve this.

(iv) Task taxonomy derived from reference SQL. The relationship-centric / analytical-heavy / local-simple taxonomy is derived from reference SQL syntax features (CTEs, window functions, etc.). This is a reasonable proxy, but it means the analysis is partly circular: it defines task difficulty in SQL terms and then finds that SQL-heavy tasks favour SQL. A model-agnostic characterisation of task complexity (e.g., from the natural-language question alone) would be more convincing.

Questions to the Authors:

  1. The workload-dependent reversal is your key finding. Have you verified that it holds for at least one other LLM? If a model with stronger Cypher proficiency narrows or eliminates the LiveSQLBench-Lite SQL advantage, the reversal may be better characterised as a model-proficiency effect than an interface–workload interaction.

  2. You report three independent runs per configuration but only aggregated accuracy. What is the standard deviation across runs for each cell in Tables 2 and 3? For the smallest deltas (e.g., +0.5 pp on BIRD Simple), is the run-to-run variance smaller than the reported effect?

  3. The graph schema is generated by PuppyGraph's built-in schema-construction agent (Section 3.1). Was the quality of this automatically generated schema validated? Poor schema construction would systematically disadvantage the graph interface in a way unrelated to the interface abstraction itself. Did you compare the auto-generated schema against a manually curated one on a subset of databases?

  4. The task taxonomy (analytical-heavy / relationship-centric / local-simple) is derived from reference-SQL features. This is a reasonable but SQL-centric proxy. Did you consider a model-agnostic alternative, e.g., classifying tasks by the number of distinct entities in the question, or by a human rating of analytical complexity, that would avoid the circularity of defining task difficulty in terms of the very language one of the interfaces uses?

  5. On LiveSQLBench-Lite, you report that "graph agents are much more likely than SQL agents to exhaust the interaction budget without producing a final executable query" (line 075). What fraction of graph failures are budget exhaustions versus incorrect but completed queries? If the graph agent simply needs more iterations for analytical composition, increasing the budget might close the gap — would this change your conclusions?

3 comments

Question 1: Generalization across language models

Response. We thank the reviewer for asking whether the workload-dependent interface pattern extends beyond GPT-5. We evaluated two additional models, DeepSeek V3.2 and Claude Haiku 5.5, using SQL and PuppyGraph on both benchmarks. The results below are relabeled accuracies.

ModelBenchmarkStructural SQLStructural PuppyGraphGraph − SQLDescriptive SQLDescriptive PuppyGraphGraph − SQL
DeepSeek V3.2BIRD55.6%57.4%+1.8 pp53.4%60.0%+6.6 pp
DeepSeek V3.2LiveSQLBench-Lite30.0%27.2%−2.8 pp27.8%22.8%−5.0 pp
Claude Haiku 5.5BIRD62.6%72.4%+9.8 pp64.6%72.6%+8.0 pp
Claude Haiku 5.5LiveSQLBench-Lite39.4%44.4%+5.0 pp43.3%42.8%−0.6 pp

On BIRD, PuppyGraph outperforms SQL for both additional models and both context settings. On LiveSQLBench-Lite, the graph advantage weakens or disappears: SQL leads for DeepSeek, while Haiku is near parity, with the direction depending on context. These results support the workload-dependent pattern across the tested models.

Question 2: Generalization across graph engines

Response. To test whether the observed pattern is specific to PuppyGraph, we evaluated two additional property-graph implementations, Neo4j and Memgraph, with DeepSeek V3.2 and Haiku 5.5 under descriptive context. The results are relabeled accuracies.

BenchmarkModelSQLPuppyGraphNeo4jMemgraph
BIRDDeepSeek V3.253.4%60.0%56.3%58.5%
BIRDHaiku 5.564.6%72.6%68.3%67.2%
LiveSQLBench-LiteDeepSeek V3.227.8%22.8%23.1%19.4%
LiveSQLBench-LiteHaiku 5.543.3%42.8%41.3%43.8%

On BIRD, all three tested graph implementations outperform SQL for both models. On LiveSQLBench-Lite, SQL outperforms all graph implementations with DeepSeek. With Haiku, SQL slightly outperforms PuppyGraph and Neo4j, while Memgraph is 0.5 percentage points above SQL. These additional results suggest that the workload-dependent pattern is not unique to PuppyGraph among the tested graph engines. We will limit our claim to the SQL and property-graph implementations evaluated here.

Questions 3 and 4: Run-to-run variation and statistical uncertainty

Response. We now report the three-run mean and sample standard deviation for each overall accuracy cell, together with the per-run Graph−SQL differences. We also computed 95% question-level paired-bootstrap intervals for the mean Graph−SQL difference. For each of 10,000 bootstrap draws, we resampled question IDs and retained the paired outcomes across all three runs.

BenchmarkContextSQL mean ± SD (%)Graph mean ± SD (%)Graph−SQL R1 / R2 / R3 (pp)Mean difference [95% paired-bootstrap CI] (pp)
BIRDStructural77.11 ± 0.5978.07 ± 0.13+0.44 / +1.33 / +1.11+0.96 [−0.59, +2.52]
BIRDDescriptive77.48 ± 0.9380.07 ± 0.56+2.89 / +3.11 / +1.78+2.59 [+1.11, +4.15]
LiveSQLBench-LiteStructural57.23 ± 0.5852.99 ± 0.67−4.62 / −2.89 / −5.20−4.24 [−8.67, +0.00]
LiveSQLBench-LiteDescriptive56.45 ± 0.8852.41 ± 1.20−2.89 / −5.20 / −4.05−4.05 [−7.71, −0.77]

The smallest BIRD differences are not clearly separated from zero: the mean Graph−SQL difference is +0.96 pp under structural context and +2.59 pp under descriptive context overall, but the structural bootstrap interval includes zero. The descriptive-context interval is above zero for BIRD and below zero for LiveSQLBench-Lite. The structural LiveSQLBench-Lite interval reaches zero. Thus, the overall reversal is supported by the paired analysis under descriptive context, while the structural-context estimates are less conclusive. These intervals are unadjusted for multiple comparisons and quantify question-level uncertainty conditional on the three observed runs; they do not estimate variation across future runs. We will report the run variation and describe the uncertainty accordingly.

Questions 5 and 6: Generated-schema validation and expert-curated comparison

Response. Having domain experts manually design and validate a production-grade schema for every benchmark database would require substantial time and specialized effort. We therefore used PuppyGraph's schema-generation agent, which has been validated by PuppyGraph to produce functional schemas similar to manually designed schemas.

We also manually inspected the generated schemas used in our experiments to verify their completeness, including checking that the benchmark-relevant tables and relationships were represented. This process let us use a scalable generation approach while checking that it did not omit required tables or relationships.

We agree that this does not directly measure how a separately expert-curated schema would change agent accuracy. We will clarify in the paper that our results concern automatically generated schemas that passed manual completeness checks; a controlled comparison against expert-curated schemas is an important direction for future work.

Question 7: Potential bias in the task taxonomy

Response. We agree that deriving task regimes from reference-SQL features makes the taxonomy SQL-centric and may introduce bias. We used this approach because it scales across the benchmark and provides a consistent, reproducible classification. Classifying complexity directly from natural-language questions or through human ratings could provide a more model-agnostic alternative, but would require substantially more annotation effort and could introduce additional judgment variation.

We will make this limitation explicit and describe the taxonomy as a proxy based on reference-query structure, rather than as a language-independent or causal measure of task difficulty.

Question 8: Graph nonfinalizations and the effect of increasing the step budget

Response. Across the two LiveSQLBench-Lite Graph contexts and three runs, the agent failed to submit a final query in 75 run-cases: 36 under structural context and 39 under descriptive context. The configured budget was 12 steps, and all 75 cases reached that limit without a final query. Across the six Graph runs there were 491 incorrect run-cases: 75 nonfinalizations (15.3%) and 416 cases in which the agent submitted a final query that was incorrect (84.7%).

The unfinished cases were concentrated in the analytical-heavy regime:

LiveSQLBench-Lite regimeStructural Graph nonfinalizationsDescriptive Graph nonfinalizationsTotal
Local/simple314
Relationship-centric459
Analytical-heavy293362
Total363975

Among the cases where Graph did not finalize, the paired SQL setting was adjudicated correct on 5 structural and 7 descriptive run-cases. These correspond to 5 of the 22 net SQL-favoring paired outcomes under structural context and 7 of the 21 under descriptive context.

We also increased the step budget for these 75 unfinished cases to 16 and 20 steps. None produced a final query; the agents continued looping. Increasing the interaction budget alone therefore did not resolve these failures or change the observed workload-dependent pattern.

Reviewer_XkeZOpenReview ↗

Comment on new experiments

Thank you, authors, for the experiments and explanation.

Although these experiments are positive and answer a few of my concerns, I view them as additional experiments to this work. I will keep my score as is; my score fits the work even with new experiments, as I had considered the probability of the extensions helping improve the work in my original score.

Reviewer_9UZtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically compares how two agent-facing database interfaces—direct SQL access and an executable semantic graph interface (Cypher over a virtual property graph mapped to the same relational data)—shape agent performance, with descriptive metadata independently controlled to separate structural interface effects from textual context. Evaluated on BIRD Mini-Dev and LiveSQLBench-Lite, the study reveals a workload-dependent performance reversal: graph interfaces outperform SQL on relationship-centric BIRD questions with gains growing with difficulty, while SQL dominates on the more analytically intensive LiveSQLBench-Lite featuring frequent multi-stage computations, window functions, and JSON operations. Further analysis shows interface choice redistributes rather than eliminates reasoning complexity: semantic structure eases data and relationship identification, while heavy analytical composition increases query-construction burden, supporting workload-aware interface design that allocates complexity between reusable system structure and agent-side query building.

Critical strengths
  1. The paper presents a methodologically sound controlled study that treats the agent-facing database interface as a first-class design choice, independently varying descriptive metadata and restricting evaluation to a shared answerable question population to eliminate confounders, avoiding the common pitfall of conflating semantic abstraction with richer textual context.
  2. It uncovers a consequential workload-dependent performance reversal between SQL and semantic graph interfaces, and further decomposes the effect into relationship-navigation and analytical-composition regimes with supporting agent-behavior evidence, delivering actionable design principles rather than merely empirical performance numbers.
  3. The evaluation is rigorous and fair, covering two benchmarks with distinct workload characteristics, three independent runs per configuration to account for agent stochasticity, and a unified two-stage adjudication pipeline that applies consistent answer-equivalence criteria across both query languages.
Critical weaknesses
  1. All experiments use only GPT-5 as the underlying agent model (Section 3.2) without validation across model families of different capabilities, so the observed workload-dependent performance reversal between interfaces remains unconfirmed for weaker or differently optimized LLMs; this is a fixable scope limitation that can be addressed by supplementing tests with additional representative models.
  2. The semantic graph schema is generated automatically by PuppyGraph’s built-in schema construction agent rather than a manually curated, production-quality semantic layer design, which may underestimate the practical advantages of well-engineered semantic abstractions and limits the real-world relevance of the graph interface findings, a fixable experimental design choice that can be improved with a professionally crafted semantic schema.
  3. The comparison is constrained to only two specific interface instantiations (relational SQL and property-graph Cypher) without covering other widely used semantic-layer paradigms such as metric layers or governed semantic query APIs, narrowing the generalizability of the paper’s core framing of database interfaces as a first-class design dimension; this is a fixable coverage gap that can be expanded by including additional interface types.
Questions for authors
  1. Since all experiments are conducted exclusively with GPT-5 as the underlying agent model, have you validated the observed workload-dependent performance reversal across LLMs of different capability levels and architectural families, to confirm the core findings are not specific to this single model?
  2. The semantic graph schema in your experiments is automatically generated by PuppyGraph’s built-in tool rather than a manually curated, production-grade semantic layer; have you assessed how a well-engineered domain-specific semantic design would change the relative performance between the two interfaces?
  3. Given that each interface excels in a distinct task regime, have you explored a hybrid interface paradigm that uses semantic graphs for relationship navigation and delegates complex analytical computation to SQL, to verify whether combining their strengths can deliver consistent performance gains across workloads?
3 comments

Question 1: Validation across language models

Response. We thank the reviewer for asking whether the observed interface pattern extends beyond our primary model. We evaluated two additional models, DeepSeek V3.2 and Claude Haiku 5.5, and report relabeled results for both SQL and PuppyGraph using the two benchmark sets. On BIRD, PuppyGraph outperforms SQL for both models and both schema conditions: by 1.8 and 6.6 percentage points for DeepSeek (structural and descriptive), and by 9.8 and 8.0 points for Haiku. On LiveSQLBench-Lite, the graph advantage is not consistent: SQL leads DeepSeek by 2.8 and 5.0 points, while Haiku is near parity overall, with PuppyGraph ahead by 5.0 points under structural context and SQL ahead by 0.6 points under descriptive context.

ModelBenchmarkStructural: SQLStructural: PuppyGraphGraph − SQLDescriptive: SQLDescriptive: PuppyGraphGraph − SQL
DeepSeek V3.2BIRD55.6%57.4%+1.8 pp53.4%60.0%+6.6 pp
DeepSeek V3.2LiveSQLBench-Lite30.0%27.2%−2.8 pp27.8%22.8%−5.0 pp
Claude Haiku 5.5BIRD62.6%72.4%+9.8 pp64.6%72.6%+8.0 pp
Claude Haiku 5.5LiveSQLBench-Lite39.4%44.4%+5.0 pp43.3%42.8%−0.6 pp

These additional results support a more qualified version of our conclusion: the graph interface advantage is consistent on BIRD across the tested models, whereas it weakens or disappears on the analytically intensive LiveSQLBench-Lite workload, with the exact direction there varying by model and schema context. These supplemental generalization results use the full benchmark sets and are reported separately from our primary paper analysis on the audited common populations.

Question 2: Automatically generated graph schema

Response. We thank the reviewer for raising the quality of the graph schema. Having domain experts manually design and validate a production-grade schema for every benchmark database would require substantial time and specialized effort. We therefore used PuppyGraph’s schema-generation agent, which has been validated by PuppyGraph to produce functional schemas similar to manually designed schemas.

We also manually inspected the generated schemas used in our experiments to verify their completeness, including checking that the benchmark-relevant tables and relationships were represented. This process let us use a scalable generation approach while checking that it did not omit required tables or relationships.

We agree that this does not directly measure how a separately expert-curated schema would change agent accuracy. We will clarify in the paper that our results concern automatically generated schemas that passed manual completeness checks; a controlled comparison against expert-curated schemas is an important direction for future work.

Question 3: Hybrid interface

Response. We thank the reviewer for this valuable suggestion. We agree that combining graph-based relationship navigation with SQL-based analytical computation is a promising direction. The goal of this paper, however, is to analyze how the choice of database interface affects agent performance; designing a fully integrated hybrid agent is beyond its scope.

We conducted an exploratory test of a simple hybrid baseline. For each question, we reused existing SQL-agent and graph-agent trajectories and provided both to a refiner, which could select one candidate answer or attempt to refine it.

DatasetGraph candidateSQL candidateOracle unionHybrid refiner
BIRD dev-mini80.7%77.8%81.8%78.2%
LiveSQLBench-Lite53.8%57.2%60.7%57.8%

On BIRD, the refiner scored below both individual candidates and the oracle union. On LiveSQLBench-Lite, it scored slightly above the SQL candidate but remained below the oracle union. Thus, this refiner did not consistently improve on the individual agents and did not reach the oracle-union result. These results suggest that simply combining two completed trajectories and asking a refiner to choose or revise an answer is not sufficient to reliably combine the strengths of both interfaces.

We view this as preliminary evidence about a simple refinement strategy, not as evidence against hybrid interfaces in general. A more principled, stage-wise hybrid agent that uses semantic graph structure for relationship navigation and SQL for analytical computation is an important direction for future work. Our present paper focuses on characterizing the interface–workload tradeoff that motivates such a design.

Critical weakness: Limited coverage of interface and graph-layer implementations

Response. We thank the reviewer for raising the generality of our findings. To test whether the observed pattern depends specifically on PuppyGraph, we evaluated two additional graph-layer implementations, Neo4j and Memgraph, using DeepSeek V3.2 and Haiku 5.5. The table reports relabeled accuracy under descriptive context.

BenchmarkModelSQLPuppyGraphNeo4jMemgraph
BIRDDeepSeek53.4%60.0%56.3%58.5%
BIRDHaiku64.6%72.6%68.3%67.2%
LiveSQLBench-LiteDeepSeek27.8%22.8%23.1%19.4%
LiveSQLBench-LiteHaiku43.3%42.8%41.3%43.8%

On BIRD, all three graph implementations outperform SQL for both tested models. On the analytically intensive LiveSQLBench-Lite benchmark, SQL outperforms all three graph implementations with DeepSeek. With Haiku, SQL slightly outperforms PuppyGraph and Neo4j, while Memgraph is 0.5 percentage points higher than SQL. Overall, the additional results support our finding that graph layers are more advantageous on BIRD, which is predominantly relationship-centric, while their advantage weakens or disappears on LiveSQLBench-Lite, which is predominantly analytical-heavy. This pattern holds across the tested graph implementations.

Thank you for the additional experiments and clarifications. Although you addressed some of my concerns, the paper's quality, topic, and preparation are still weak, so I prefer to keep the original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.