acceptodds
Under review as a conference paper at ICLR 2027

BIRD-Pandas: Diagnosing Cross-Paradigm Divergence in Text-to-SQL and Text-to-Pandas

Abstract

Text-to-SQL is a widely adopted paradigm for natural-language access to structured data, but many real-world analytics tasks involve file-based tables and multi-step transformations that are more naturally handled with code-based approaches. Controlled evaluation of Text-to-Pandas and Text-to-SQL under matched task conditions remains largely absent, making it difficult to isolate paradigm-specific challenges from task content. We introduce BIRD-Pandas, a paired benchmark that derives verified Pandas ground truth from BIRD and enables controlled comparison. Benchmark construction revealed that producing equivalent results across paradigms requires different decision logic due to differences in how the two paradigms allocate execution decisions, a phenomenon we term execution-semantic divergence. Text-to-Pandas trails Text-to-SQL in accuracy at smaller and mid-scale models, and this gap narrows substantially as model scale increases. However, the aggregate gap reflects multiple contributing factors, including execution-semantic divergence and task-specification insufficiency, where natural language questions omit information. Disentangling these factors requires supplying the missing task information and observing the resulting performance change. To this end, we introduce the Logic Completion Framework (LCF), a clarification protocol that measures performance recovery when the model requests and receives such information. LCF closes most of the cross-paradigm gap at intermediate scales. At the largest evaluated scale, SQL and Pandas achieve statistically comparable aggregate accuracy, yet SQL benefits more than Pandas from complete task specification. Analysis of post-LCF failures shows that errors attributable to task-specification insufficiency remain within both paradigms' own error sets, as models tend to overlook consequential ambiguities even when invited to ask clarifying questions, while the net cross-paradigm gap is instead driven mainly by execution-semantic divergence, which is concentrated on the Pandas side. Resources are available at https://anonymous.4open.science/r/Bird_Pandas-3F82/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
text2sql, code generation, data query

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sKbwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose BIRD-Pandas, a new paired benchmark for comparing Text-to-SQL and Text-toPandas generations. The benchmark is constructed from the BIRD development set by correcting reference SQL queries and deriving equivalent Pandas programs.

The authors investigate two potential sources of performance differences: execution-semantic divergence, arising from differences in execution semantics and API behavior, and task-specification insufficiency, arising from missing or ambiguous task information. They also introduce the Logic Completion Framework (LCF), which uses a ground-truth-informed oracle to provide clarifications before program generation.

Critical strengths

S1. The paire evaluation design provide a baseline for comparing the performance of model on SQL and pandas/dataframe generation.

S2. Constructing the benchmark itself involves substantial labours and efforts.

S3. The experiments cover multiple model families and scales.

Critical weaknesses

W1. The necessity of constructing a dedicated Text-to-Pandas benchmark remains questionable. While evaluating natural-language-to-DataFrame generation is a reasonable research direction, the paper does not sufficiently explain why it requires a separate benchmark derived entirely from existing Text-to-SQL tasks. In particular, all tasks in BIRD-Pandas are originally SQL-expressible, and existing systems such as DuckDB and Ibis already enable SQL-based or relational-expression-based processing over DataFrames. Therefore, the practical motivation for directly generating Pandas code, rather than leveraging existing relational abstractions, is not clearly established.

W2. The paper does not clearly establish which fundamentally different reasoning capabilities are required by Text-to-Pandas compared with Text-to-SQL. Core challenges such as intent understanding, schema linking, entity resolution, join reasoning, and query planning are largely shared between the two paradigms. For the relational workloads considered in this benchmark, the primary distinction lies in the target programming language and its execution semantics, rather than the underlying data reasoning problem. Although SQL and Pandas differ in their APIs and default behaviors, it remains unclear whether these differences justify a separate benchmark or reveal a sufficiently important new capability of LLMs.

W3. The paper lacks comparisons with straightforward alternatives that could potentially address the same tasks without directly generating Pandas code. For example, an LLM could first generate SQL and then execute it over DataFrames using DuckDB, or leverage a relational intermediate representation such as Ibis. These alternatives would reuse existing Text-to-SQL capabilities while avoiding many of the API-specific difficulties highlighted in the paper. Without evaluating such baselines, it is difficult to determine whether the observed Text-to-Pandas failures represent fundamental limitations of LLM-based data analysis or simply additional complexity introduced by directly generating Pandas operations. Such comparisons are important for establishing both the necessity and the practical value of the proposed benchmark.

Questions for authors

Q1. What important real-world capabilities does BIRD-Pandas evaluate that cannot be addressed through Text-to-SQL followed by execution over DataFrames? Can the authors provide a baseline using DuckDB, Ibis, or another relational intermediate representation?

3 comments

We thank the reviewer for the constructive feedback. Below we clarify the scope of our study and the limitations of our evaluation.

R1 and Q1: Our motivation is to evaluate a different target interface on matched tasks: given the same question and underlying data, how reliably can an LLM generate direct Pandas operations, and how does this compare with SQL generation?

Starting from BIRD makes this comparison possible while reducing differences in task content. BIRD-Pandas adds verified Pandas reference programs paired with SQL-derived answers, providing a reusable resource for studying cross-paradigm behavior. This is a controlled benchmark for the shared querying workload, rather than an evaluation of the full range of Pandas workflows.

R2: We agree that the underlying data reasoning is largely shared, and we do not claim a fundamentally different reasoning capability. Our contribution is to examine whether generating programs for different target interfaces introduces additional, measurable requirements even on the same querying tasks.

For example, in Figure 1, SQL GROUP BY retains NULL-key groups, whereas Pandas groupby drops them by default. When the intended answer includes those groups, the Pandas program must explicitly account for that behavior. Such cases illustrate why correct task reasoning alone may not be sufficient for correct program execution.

During benchmark construction, 370 SQL-to-Pandas conversion instances required manual intervention. These cases include different types of mismatch and should not all be interpreted as evidence of API-default errors. The paired benchmark enables systematic study of such differences, while remaining limited to BIRD-derived relational tasks.

R3 and Q1: Following the reviewer’s suggestion, we evaluated DuckDB on all 1,534 instances for three Qwen3 models. We reused the SQL predictions from the Verified SQLite condition and executed them over Pandas DataFrames with DuckDB, either directly (raw) or after SQLite-to-DuckDB dialect translation with SQLGlot (transpiled).

ModelPandasSQL (SQLite)SQL via DuckDB (transpiled)SQL via DuckDB (raw)
Qwen3-8B53.1961.2159.0056.58
Qwen3-32B60.2361.8059.9157.56
Qwen3-Max63.4363.1060.6956.19

In this configuration, both DuckDB variants score below the original SQLite condition. They outperform direct Pandas for Qwen3-8B but score lower for Qwen3-32B and Qwen3-Max. This comparison uses SQLite-targeted SQL, rather than SQL generated specifically for DuckDB; dialect translation, engine behavior, and DataFrame loading may affect the results. We therefore do not claim that direct Pandas is generally superior.

We agree that Text-to-SQL followed by DuckDB execution is a practical alternative. However, the two methods evaluate different targets: DuckDB executes generated SQL over DataFrames, whereas BIRD-Pandas measures the reliability of generating direct Pandas operations. The latter is relevant when a workflow requires Pandas code, and the paired benchmark allows us to study target-specific requirements under matched tasks.

The Introduction, Related Work and Limitations now distinguish these routes from direct Pandas generation, and the appendix reports the DuckDB reference.

We sincerely thank reviewers for the time, care, and expertise they have devoted to evaluating our work. We have taken the feedback seriously and prepared detailed responses to every concern. These revisions reflect our careful effort to make the work more rigorous, transparent, and useful to the community.

We respectfully hope that the reviewers will have an opportunity to consider our responses. If the clarifications address the concerns raised, we would be truly grateful if you would feel comfortable reconsidering the rating. We remain fully available for any further questions or discussion during the rebuttal period.

Dear Reviewer sKbw,

We thank you again for the time and expertise devoted to evaluating our work. We have responded to each of the points raised and revised the manuscript accordingly. If these responses resolve your concerns, we would be grateful if you would consider adjusting the rating where appropriate. If any questions remain, we would welcome further discussion.

Best regards, The Authors

Dear Authors,

I appreciate your efforts to strengthen the paper. The manuscript is already well-written and comprehensive. However, my primary concern remains unresolved: whether a DataFrame-based approach is truly necessary for problems that can already be addressed using SQL, and whether it is necessary to adapt a SQL-based benchmark for it. And, it seems for larger model the gap between DataFrame-base approach and SQL-based approach seems to shrinking.

As this fundamental concern has not been sufficiently addressed, I have decided to maintain my original rating.

Reviewer_ki6BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BIRD-Pandas, a paired benchmark derived from the 1,534-instance BIRD development set for comparing Text-to-SQL and Text-to-Pandas. The construction pipeline corrects 259 SQL annotations and manually resolves 370 SQL-to-Pandas conversion cases. The authors evaluate multiple language models using an LLM-based output-equivalence validator and introduce the Logic Completion Framework (LCF), a single-turn, ground-truth-assisted clarification protocol. The reported SQL–Pandas gap becomes small for several stronger models under the standard setting. LCF substantially improves both paradigms and produces near-equal aggregate accuracy for the models labeled Qwen3-7B and Qwen3-32B, while Qwen3-Max reaches 80.05% for SQL and 73.21% for Pandas. The paper attributes much of this remaining difference to execution-semantic divergence through residual-error analysis, while also reporting a human validator audit and paired statistical tests.

Critical strengths
  1. BIRD-Pandas aligns 1,534 tasks across SQL and Pandas, with 259 SQL annotation corrections and manual resolution of 370 conversion cases. This combination of paired evaluation and reference curation represents a valuable contribution to investigating differences between the two paradigms.
  2. LCF offers a structured way to examine how clarification changes performance, complemented by a 600-instance validator audit and paired statistical tests. These analyses strengthen the empirical contribution by showing where aggregate performance converges and where instance-level disagreement remains, providing useful evidence beyond a single benchmark score.
Critical weaknesses
  1. The text describes an output-only validator, but Table 11 also supplies the reference query; its multiset comparison rule conflicts with unconditional deduplication. These inconsistencies can change correctness labels and therefore the reported SQL–Pandas gap. Without a consistent deployed specification and sensitivity checks, I cannot confidently assess the central empirical comparison.
  2. One clarification round does not establish complete or identical task information across paradigms, yet the paper attributes the residual gap strongly to execution semantics. Appendix A itself describes this attribution as a diagnostic upper bound. The claim should be narrowed or supported by stronger evidence.
Questions for authors
  1. Table 11 requires multiset comparison in Rule 3 but removes duplicates in Rule 7. Which rule was actually applied when duplicate rows changed the answer? Please clarify this conflict and report whether adjudicating the affected cases under the intended task semantics changes the SQL–Pandas gap; the overall agreement rate in Table 23 does not resolve this specific issue.
  2. In Table 1, TableGPT2’s verified Pandas scores are 25.46%, 17.24%, and 13.10%. Weighting these by the BIRD dev counts of 925, 464, and 145 gives 21.81%, rather than the reported 19.82%. Please reconcile this discrepancy and indicate whether correcting the affected entries changes the reported comparisons between models and paradigms. Similar discrepancies appear in other rows (e.g., AutoCoder and StarCoder2). Please verify all aggregate scores in Table 1.
  3. Appendix A acknowledges that missed clarification questions can inflate the execution-semantic category, yet the main text attributes the residual gap strongly to execution semantics. What evidence distinguishes incomplete clarification from execution-semantic errors within the 21.9% category in Table 3? This distinction is necessary to assess the explanatory claim beyond the statistically significant accuracy gap.
  4. Section 4.3 reports Pandas gains of 8.15 and 6.78 percentage points, but these accuracy shifts do not establish reduced annotation-induced variance. Were the same predictions re-scored against the original and verified references? Please clarify whether the evidence supports a change in measured accuracy or an actual reduction in measurement variance, and revise the claim accordingly.
2 comments

We thank the reviewer for the constructive feedback. Below we clarify the scope of our study and the limitations of our evaluation.

R1: Thank you for identifying this inconsistency. The validator compares the predicted and reference execution outputs, with the reference SQL query supplied as context. Our previous description as “output-only” was misleading.

Rule 3 and Rule 7 apply to different situations. Rule 3 sets the default comparison, which ignores row order unless the reference query contains ORDER BY or LIMIT. Rule 7 applies when only one output contains duplicate rows. When duplicate rows change the answer, Rule 7 is therefore the rule that applies.

To test whether adjudicating under the intended task semantics changes the SQL-Pandas gap, we re-adjudicated, on Qwen2.5-Coder (14B), every case in both paradigms whose output differs from the reference only in duplicate rows or only in row order. Only 7 labels were affected (6 Pandas, 1 SQL). Pandas accuracy moves from 54.30% to 54.69% and SQL accuracy from 58.74% to 58.67%, so the gap moves from 4.44 to 3.98 percentage points. Re-adjudication under the intended task semantics therefore does not change the SQL-Pandas gap. We have added the adjudicated labels and this check to the appendix.

R2: The Totals for StarCoder2, AutoCoder and TableGPT2 in Table 1 were produced incorrectly in the aggregation step that generates the Total column. The per-instance correctness judgments and the difficulty-level results are unaffected. We recomputed each affected Total as the sum of the correct counts over the three difficulty levels divided by all 1,534 instances, and confirmed that the other rows of Table 1 already satisfy this relation. The corrected Totals and the resulting SQL-Pandas gaps (SQL minus Pandas) are given below, with the previously reported values in parentheses.

ModelSettingPandas Total (was)SQL TotalSQL-Pandas gap (was)
StarCoder2Verified1.43 (1.12)0.91−0.52 (−0.21)
StarCoder2Original1.11 (0.85)0.72−0.39 (−0.13)
AutoCoderVerified6.71 (5.45)10.233.52 (4.78)
AutoCoderOriginal5.80 (4.88)9.193.39 (4.31)
TableGPT2Verified21.77 (19.82)34.0312.26 (14.21)
TableGPT2Original19.56 (17.45)32.5913.03 (15.14)

The corrections do not change the comparisons between models or between paradigms. Only the Pandas Totals of these three models change. Between models, the three remain the lowest-performing group under both paradigms and keep their relative order (StarCoder2, AutoCoder, TableGPT2). Between paradigms, the direction of each gap is unchanged: SQL remains ahead for AutoCoder and TableGPT2, and for StarCoder2 the two paradigms remain within one point of each other, with Pandas nominally ahead. The gaps for AutoCoder and TableGPT2 narrow by about 0.9 to 2.1 percentage points. We have updated Table 1 in the manuscript.

R3: We thank the reviewer for pointing this out. We have revised the main text so that the residual gap after clarification is no longer strongly attributed to execution-semantic divergence. Within the 21.9% category, the current design cannot separate incomplete clarification from execution-semantic errors, because LCF answers only the questions the model itself raises. An ambiguity the model fails to raise stays unresolved, and the resulting error may be counted as execution-semantic divergence. Conversely, some execution-semantic effects may be recorded as Phase 3 code-generation errors, and classification noise can shift labels in either direction. We therefore report the 21.9% as an estimate under our protocol, subject to these limitations. We have revised the Limitations accordingly. Separating the two causes requires a separate study with a redesigned framework, for example Oracle-driven exhaustive clarification.

R4: The same predictions were re-scored against the original and the verified references, so the two rows per model in Table 1 differ only in the reference used. The gains therefore reflect a change in measured accuracy, not a reduction in measurement variance. "Variance" was imprecise wording, and we have replaced it in Section 4.3 with "annotation-induced divergence", meaning the difference in measured accuracy caused by the reference annotations.

We sincerely thank reviewers for the time, care, and expertise they have devoted to evaluating our work. We have taken the feedback seriously and prepared detailed responses to every concern. These revisions reflect our careful effort to make the work more rigorous, transparent, and useful to the community.

We respectfully hope that the reviewers will have an opportunity to consider our responses. If the clarifications address the concerns raised, we would be truly grateful if you would feel comfortable reconsidering the rating. We remain fully available for any further questions or discussion during the rebuttal period.

Dear Reviewer ki6B,

We thank you again for the time and expertise devoted to evaluating our work. We have responded to each of the points raised and revised the manuscript accordingly. If these responses resolve your concerns, we would be grateful if you would consider adjusting the rating where appropriate. If any questions remain, we would welcome further discussion.

Best regards, The Authors

Reviewer_9XuNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work performed a controlled analysis on text-to-pandas and text-to-sql. The authors introduced a paired benchmark with both pandas and SQL on the same underlying data and constraints, derived from BIRD. Then the authors introduced the Logic Completion Framework to measure the performance differences and subsequently presented insights on the paradigm gaps.

Critical strengths

S1: The proposed paired benchmark introduces a controlled setup to compare text-to-pandas and text-to-sqly directly.

S2: The authors evaluated multiple models with different sizes and training paradigms, as well as open-source and proprietary.

S3: The derived insights are interesting: pandas and SQL achieve comparable performance when the user intents are clarified.

Critical weaknesses

W1: The current setup for analysis is narrowed down to a very specific angle, and in real-world scenarios there might be other capabilities, such as a retrieval step, where one is much more capable than the others.

W2: The evaluation is done on the clean data on BIRD benchmark, which is not the best candidate to evaluate Python’s capability to handle ML integration, etc. Furthermore, the benchmark doesn’t reflect realistic scenarios even for SQL’s analytical capabilities.

W3: The evaluation is done entirely on the execution accuracy, which again does not necessarily reflect how good the code is from both Python and SQL.

Questions for authors

W1-W3

2 comments

We thank the reviewer for the constructive feedback. Below we clarify the scope of our study and the limitations of our evaluation.

R1: We agree that retrieval is an important component of real analytical workflows and may affect the relative performance of SQL and Pandas systems. Our goal is narrower: to compare program generation under matched questions, declared schemas, external knowledge, and underlying data, without an additional retrieval stage. This helps reduce differences in task information when studying cross-paradigm behavior. As discussed in Related Work, open-ended settings such as DaBStep and Spider 2.0 couple information retrieval with core reasoning, which suits the research questions they study. Our question is different, and we do not claim that our findings transfer to pipelines with a retrieval step. We have added a discussion to the Limitations section of the revised manuscript to clarify the scope of our study with respect to retrieval.

R2: We agree that BIRD-Pandas does not evaluate the full range of Python-based analytics, including machine-learning integration or open-ended data cleaning. Our focus is the narrower set of data-querying tasks for which SQL and Pandas can be compared on matched questions and data. BIRD does not provide paired tasks covering the broader ML workflow, so evaluating that capability would require a different task design, which is not our purpose of this paper. On realism, BIRD was designed to narrow the gap between academic study and real-world applications: it is built on open-source databases from real analysis platforms and targets dirty and noisy database values, external knowledge grounding, and SQL efficiency on massive databases. In BIRD-dev, about 14% of cells are NULL, 1.7% are empty strings, and 49% of columns contain at least one missing value; format issues also exist, e.g., numeric columns with inequality-prefixed values (such as "<0.3"), which to some extent reflects real-world data conditions. BIRD-Pandas evaluates both paradigms on the questions and databases of BIRD.

R3: We agree that execution accuracy does not capture all aspects of code quality, such as readability, maintainability, or efficiency. Our goal, however, is to compare models’ ability to generate programs that answer the same data-querying questions under SQL and Pandas, rather than to evaluate overall code quality. For this purpose, we assess whether the executed program produces an answer equivalent to the verified reference. Execution-based evaluation is widely used in Text-to-SQL work, such as Seq2SQL, Spider, and BIRD, and we apply a shared equivalence criterion to both paradigms.

We sincerely thank reviewers for the time, care, and expertise they have devoted to evaluating our work. We have taken the feedback seriously and prepared detailed responses to every concern. These revisions reflect our careful effort to make the work more rigorous, transparent, and useful to the community.

We respectfully hope that the reviewers will have an opportunity to consider our responses. If the clarifications address the concerns raised, we would be truly grateful if you would feel comfortable reconsidering the rating. We remain fully available for any further questions or discussion during the rebuttal period.

Dear Reviewer 9XuN,

We thank you again for the time and expertise devoted to evaluating our work. We have responded to each of the points raised and revised the manuscript accordingly. If these responses resolve your concerns, we would be grateful if you would consider adjusting the rating where appropriate. If any questions remain, we would welcome further discussion.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.