- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper compares a GPT-5 database agent using direct PostgreSQL and SQL with the same agent using a virtual semantic graph queried through Cypher and PuppyGraph. Descriptive metadata is varied separately within each interface. On a common-answerable set of 450 BIRD Mini-Dev and 173 LiveSQLBench-Lite questions, the graph setting leads by 1.0–2.6 points on BIRD, while SQL leads by 4.1–4.2 points on LiveSQLBench-Lite. The proposed explanation is that executable relationships help when navigation dominates, whereas constructing multi-stage analytical queries remains harder through the graph interface.
Critical strengths
- The study treats the executable database interface as a first-class design choice. It holds the underlying data and agent model fixed, varies descriptive metadata within both interfaces, and audits graph answerability on the retained evaluation sets.
- The paper goes beyond an overall win rate. Its workload taxonomy, stratified results, and query-finalization traces offer a coherent account of why the preferred interface might reverse between relationship-centric and analytically heavy tasks.
Critical weaknesses
- The smaller interface differences are difficult to assess from the reported aggregates. Each condition is run three times, but the paper omits per-run scores and paired Graph–SQL outcomes. This is particularly consequential for the 1.0- and 2.6-point BIRD advantages. Rule-based incorrect verdicts can also be overturned by an LLM judge; the same adjudication policy is applied to both interfaces, but overturn counts by condition and scores before adjudication are not reported. Without these, a reader cannot tell whether the direction and size of the smaller effects are stable or sensitive to evaluation decisions.
- The workload account is plausible but not yet demonstrated across the reported runs. The regime contributions are explicitly computed for an analyzed round, and their sums differ from the three-run aggregate interface gaps. The paper does not identify that round or show whether the claimed relationship-centric and analytical-heavy contributions persist across runs. Likewise, the 75 graph-agent failures to finalize a query on LiveSQLBench-Lite are suggestive, but their distribution across task regimes and contribution to the SQL advantage are not quantified. These observations support a hypothesis about this SQL/Cypher/PuppyGraph comparison, not yet a general conclusion about semantic interfaces.
Questions for authors
- From the existing runs, can you report per-run Graph–SQL differences and paired question-level outcomes, along with adjudication overturn counts and overall scores before and after adjudication for each condition? Does the interface preference hold in every run and under rule-based scoring alone?
- Which run underlies the task-regime contribution analysis, and do the same dominant contributions appear in the other runs? Among the 75 graph-agent nonfinalizations, how many occur in each task regime, and how much of the observed SQL advantage do they account for?
3 comments
Example of a rule-based false negative
In BIRD Graph-descriptive Round 1, the question asks for valid administrator email addresses for a school matching specified conditions. The strict scorer marks the answer incorrect because the gold and Graph outputs have different shapes, even though they contain the same two requested addresses.
Gold result
admemail1 | admemail2 |
|---|---|
| [email protected] | [email protected] |
Graph result
admin_email |
|---|
| [email protected] |
| [email protected] |
LLM-based adjudication recognizes that the two requested email values and the filtering conditions match, so it marks this answer correct. The mismatch is only in presentation: two columns in one gold row versus one column with two Graph rows.
Question 2: Task-regime analysis and unfinished queries
Response. We thank the reviewer for pointing out that the paper did not identify the run used for its task-regime analysis or report how unfinished queries were distributed across regimes. The manuscript draft indicates that the reported regime analysis was intended to use Round 2. That table reflects a single-round analysis on the subset used at the time, which excluded unfinished cases and other cases under the then-used filtering procedure. We did not retain a complete item-level record of those exclusions, so we cannot reconstruct that exact subset.
For a more transparent comparison across runs, we now report the weighted Graph–SQL effects for all three rounds on the full audited valid set: 450 BIRD questions and 173 LiveSQLBench-Lite questions. We retain unfinished cases and count them as incorrect. Values are weighted contributions in percentage points; the three-round mean is the arithmetic mean of the round-level values.
| Benchmark | Task regime | Structural R1 | R2 | R3 | Mean | Descriptive R1 | R2 | R3 | Mean |
|---|---|---|---|---|---|---|---|---|---|
| BIRD | Local/simple | +0.22 | +0.00 | +0.22 | +0.15 | +0.44 | +0.22 | +0.67 | +0.44 |
| BIRD | Relationship-centric | +0.22 | +1.11 | +0.22 | +0.52 | +2.00 | +2.67 | +1.11 | +1.93 |
| BIRD | Analytical-heavy | +0.00 | +0.22 | +0.67 | +0.30 | +0.44 | +0.22 | +0.00 | +0.22 |
| LiveSQLBench-Lite | Local/simple | −1.73 | −1.16 | −0.58 | −1.16 | −0.58 | −0.58 | −0.58 | −0.58 |
| LiveSQLBench-Lite | Relationship-centric | −1.16 | +0.00 | −1.16 | −0.77 | +0.00 | −0.58 | −0.58 | −0.39 |
| LiveSQLBench-Lite | Analytical-heavy | −1.73 | −1.73 | −3.47 | −2.31 | −2.31 | −4.05 | −2.89 | −3.08 |
The unfinished-query counts below are run-cases across the three rounds, not unique questions. The 75 LiveSQLBench-Lite Graph nonfinalizations are concentrated in the analytical-heavy regime. Among these cases, the paired SQL setting was adjudicated correct in 5 structural and 7 descriptive cases. These cases account for 5 of the 22 net SQL-favoring paired outcomes in structural runs and 7 of the 21 in descriptive runs; the remaining outcomes reflect the other paired questions.
| LiveSQLBench-Lite regime | Graph structural unfinished | Graph descriptive unfinished | Graph total | SQL correct on paired Graph unfinished: structural | SQL correct on paired Graph unfinished: descriptive |
|---|---|---|---|---|---|
| Local/simple | 3 | 1 | 4 | 0 | 0 |
| Relationship-centric | 4 | 5 | 9 | 0 | 0 |
| Analytical-heavy | 29 | 33 | 62 | 5 | 7 |
| Total | 36 | 39 | 75 | 5 | 7 |
The three-round results show a consistent workload-level pattern: relationship-centric regimes favor Graph on BIRD, while analytical-heavy tasks favor SQL on LiveSQLBench-Lite. We will clarify the single-round nature of the original regime analysis and add the three-round results and unfinished-query breakdown.
Thank you for the additional experiments and clarifications. They address some of my concerns, so I am raising my soundness rating to 3.
Question 1: Per-run results, paired outcomes, and LLM-based adjudication
Response. We thank the reviewer for asking us to report run-level results and the effect of adjudication. We compare strict rule-based execution accuracy with accuracy after the paper's LLM-based adjudication. Strict result comparison can mark an answer incorrect when it contains the requested information in a different output shape. LLM-based adjudication applies the same conservative answer-equivalence policy to SQL and Graph; it does not excuse differences in filters, join paths, aggregation grain, denominators, or requested answer content.
The first table answers two parts of the reviewer's question: how each condition scores before and after LLM-based adjudication in each run, and how many strict false negatives are reclassified for each condition. BIRD values are the aligned run views in the clean-results manifest, including documented targeted replacements; LiveSQLBench-Lite values use the aligned 173-question valid-case runs. Scores are shown as
correct / valid questions (accuracy). Reclassification totals count run-cases across three runs, not unique questions.This table shows substantial adjudication gains in every condition, with the largest gains for Graph on LiveSQLBench-Lite. These are not arbitrary score increases: the LLM-based adjudication corrects strict-match false negatives when the answer is semantically equivalent but represented differently. No rule-based correct case was overturned as incorrect in these aligned views.
The second table answers whether the Graph–SQL difference persists run by run and reports the paired question-level outcomes.
Graph-onlyandSQL-onlycount questions answered correctly by just one interface in that paired run.Graph−SQLis their net difference expressed in percentage points.The paired results show that the preference is stable across all three runs and under both scoring procedures: Graph leads on BIRD, while SQL leads on LiveSQLBench-Lite. Thus, the direction of the workload-dependent finding is not introduced by LLM-based adjudication. Adjudication primarily changes the estimated size of the gap, substantially narrowing the strict-score SQL advantage on LiveSQLBench-Lite.