- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper propose BIRD-Pandas, a new paired benchmark for comparing Text-to-SQL and Text-toPandas generations. The benchmark is constructed from the BIRD development set by correcting reference SQL queries and deriving equivalent Pandas programs.
The authors investigate two potential sources of performance differences: execution-semantic divergence, arising from differences in execution semantics and API behavior, and task-specification insufficiency, arising from missing or ambiguous task information. They also introduce the Logic Completion Framework (LCF), which uses a ground-truth-informed oracle to provide clarifications before program generation.
Critical strengths
S1. The paire evaluation design provide a baseline for comparing the performance of model on SQL and pandas/dataframe generation.
S2. Constructing the benchmark itself involves substantial labours and efforts.
S3. The experiments cover multiple model families and scales.
Critical weaknesses
W1. The necessity of constructing a dedicated Text-to-Pandas benchmark remains questionable. While evaluating natural-language-to-DataFrame generation is a reasonable research direction, the paper does not sufficiently explain why it requires a separate benchmark derived entirely from existing Text-to-SQL tasks. In particular, all tasks in BIRD-Pandas are originally SQL-expressible, and existing systems such as DuckDB and Ibis already enable SQL-based or relational-expression-based processing over DataFrames. Therefore, the practical motivation for directly generating Pandas code, rather than leveraging existing relational abstractions, is not clearly established.
W2. The paper does not clearly establish which fundamentally different reasoning capabilities are required by Text-to-Pandas compared with Text-to-SQL. Core challenges such as intent understanding, schema linking, entity resolution, join reasoning, and query planning are largely shared between the two paradigms. For the relational workloads considered in this benchmark, the primary distinction lies in the target programming language and its execution semantics, rather than the underlying data reasoning problem. Although SQL and Pandas differ in their APIs and default behaviors, it remains unclear whether these differences justify a separate benchmark or reveal a sufficiently important new capability of LLMs.
W3. The paper lacks comparisons with straightforward alternatives that could potentially address the same tasks without directly generating Pandas code. For example, an LLM could first generate SQL and then execute it over DataFrames using DuckDB, or leverage a relational intermediate representation such as Ibis. These alternatives would reuse existing Text-to-SQL capabilities while avoiding many of the API-specific difficulties highlighted in the paper. Without evaluating such baselines, it is difficult to determine whether the observed Text-to-Pandas failures represent fundamental limitations of LLM-based data analysis or simply additional complexity introduced by directly generating Pandas operations. Such comparisons are important for establishing both the necessity and the practical value of the proposed benchmark.
Questions for authors
Q1. What important real-world capabilities does BIRD-Pandas evaluate that cannot be addressed through Text-to-SQL followed by execution over DataFrames? Can the authors provide a baseline using DuckDB, Ibis, or another relational intermediate representation?
3 comments
Dear Reviewer sKbw,
We thank you again for the time and expertise devoted to evaluating our work. We have responded to each of the points raised and revised the manuscript accordingly. If these responses resolve your concerns, we would be grateful if you would consider adjusting the rating where appropriate. If any questions remain, we would welcome further discussion.
Best regards, The Authors
Dear Authors,
I appreciate your efforts to strengthen the paper. The manuscript is already well-written and comprehensive. However, my primary concern remains unresolved: whether a DataFrame-based approach is truly necessary for problems that can already be addressed using SQL, and whether it is necessary to adapt a SQL-based benchmark for it. And, it seems for larger model the gap between DataFrame-base approach and SQL-based approach seems to shrinking.
As this fundamental concern has not been sufficiently addressed, I have decided to maintain my original rating.
We thank the reviewer for the constructive feedback. Below we clarify the scope of our study and the limitations of our evaluation.
R1 and Q1: Our motivation is to evaluate a different target interface on matched tasks: given the same question and underlying data, how reliably can an LLM generate direct Pandas operations, and how does this compare with SQL generation?
Starting from BIRD makes this comparison possible while reducing differences in task content. BIRD-Pandas adds verified Pandas reference programs paired with SQL-derived answers, providing a reusable resource for studying cross-paradigm behavior. This is a controlled benchmark for the shared querying workload, rather than an evaluation of the full range of Pandas workflows.
R2: We agree that the underlying data reasoning is largely shared, and we do not claim a fundamentally different reasoning capability. Our contribution is to examine whether generating programs for different target interfaces introduces additional, measurable requirements even on the same querying tasks.
For example, in Figure 1, SQL GROUP BY retains NULL-key groups, whereas Pandas groupby drops them by default. When the intended answer includes those groups, the Pandas program must explicitly account for that behavior. Such cases illustrate why correct task reasoning alone may not be sufficient for correct program execution.
During benchmark construction, 370 SQL-to-Pandas conversion instances required manual intervention. These cases include different types of mismatch and should not all be interpreted as evidence of API-default errors. The paired benchmark enables systematic study of such differences, while remaining limited to BIRD-derived relational tasks.
R3 and Q1: Following the reviewer’s suggestion, we evaluated DuckDB on all 1,534 instances for three Qwen3 models. We reused the SQL predictions from the Verified SQLite condition and executed them over Pandas DataFrames with DuckDB, either directly (raw) or after SQLite-to-DuckDB dialect translation with SQLGlot (transpiled).
In this configuration, both DuckDB variants score below the original SQLite condition. They outperform direct Pandas for Qwen3-8B but score lower for Qwen3-32B and Qwen3-Max. This comparison uses SQLite-targeted SQL, rather than SQL generated specifically for DuckDB; dialect translation, engine behavior, and DataFrame loading may affect the results. We therefore do not claim that direct Pandas is generally superior.
We agree that Text-to-SQL followed by DuckDB execution is a practical alternative. However, the two methods evaluate different targets: DuckDB executes generated SQL over DataFrames, whereas BIRD-Pandas measures the reliability of generating direct Pandas operations. The latter is relevant when a workflow requires Pandas code, and the paired benchmark allows us to study target-specific requirements under matched tasks.
The Introduction, Related Work and Limitations now distinguish these routes from direct Pandas generation, and the appendix reports the DuckDB reference.
We sincerely thank reviewers for the time, care, and expertise they have devoted to evaluating our work. We have taken the feedback seriously and prepared detailed responses to every concern. These revisions reflect our careful effort to make the work more rigorous, transparent, and useful to the community.
We respectfully hope that the reviewers will have an opportunity to consider our responses. If the clarifications address the concerns raised, we would be truly grateful if you would feel comfortable reconsidering the rating. We remain fully available for any further questions or discussion during the rebuttal period.