acceptodds
Under review as a conference paper at ICLR 2027

SemBench-Hard: Evaluating Query Translation and Operator Grounding in LLM-Backed Semantic Query Processing

Abstract

Structured LLM workflows must turn a request into steps, and each intermediate step must return not only a decision but a grounded artifact that later steps can check and consume. Semantic query-processing engines (SQPEs) expose this need: they answer analytics questions over text-rich records with LLM-backed FILTER, JOIN, MAP, and RANK operators. An NL-facing SQPE can fail in two places: when it compiles a question into an operator program, and when an operator returns a decision without the grounding, such as evidence sentences or attributes, that its output contract requires. SemBench-Hard makes both harder and evaluates them separately. Its translation axis withholds the step-by-step plan from the question and scores generated programs by their final output; its execution axis runs reference programs through three system adapters with three backbone LLMs on paired Base and Hard operator tasks over four public datasets, where Hard adds distractors that make the right grounding harder to find, explicit constraints, output schemas, and deterministic grounding checks. On the translation axis, many accepted programs differ from the reference, so exact-match program scoring is too strict, and goal-only queries lead two of four translators to treat MAP’s string output as a structured object. On the execution axis, Hard tasks lower strict (reference-exact) acceptance from 86.0% to 60.1% (−25.9 points); crediting grounding-only rejections at the rate human annotators judge broadly valid leaves −16.1, and accepting every grounding-only rejection leaves −6.3. The main finding is that operators mostly keep the decision but often return grounding that fails the contract’s check; annotators judge about a third of these rejected outputs insufficiently grounded or wrong, three times the rate among accepted outputs. The loss persists: chains (LOTUS × Qwen3) stay 25–31 points lower under Hard at every length, and bounded task-local tools add 1.0 point at 2.24 times the API cost (4.0 with a SWE-bench helper). SQPEs should therefore schema-check generated programs and check each operator’s grounding.

Primary area:
datasets and benchmarks
Keywords:
semantic query processing, LLM-backed operators, benchmarks, natural-language-to-program translation, verifier-grounded execution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HcyeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SemBench-Hard, a benchmark for natural-language query translation and the execution of LLM-backed semantic operators. It evaluates whether operators return not just correct decisions but also the evidence or structured outputs needed by downstream steps. Across multiple datasets, systems, and models, the authors find that grounding requirements reveal substantial failures that decision-only evaluation would miss.

Critical strengths

The paper addresses a practical and underexplored problem. I like the separation between query translation and operator execution, and especially the focus on whether intermediate outputs are actually usable. The experiments cover a good range of settings, and the human evaluation and ablations help substantiate the main observation rather than relying on one benchmark score.

Critical weaknesses

My main concern is how the Base–Hard gap is interpreted. Hard tasks change both the input and the success criteria, so the headline drop does not isolate a decline in semantic reasoning. The relaxed and human-calibrated grounding results deserve at least as much emphasis as strict acceptance. The translation results are also harder to interpret because the reference programs themselves have fairly low success rates under the primary executor.

Questions for authors

Could the authors more clearly separate decision errors from failures of the grounding contract? Also, how much of the translation trend remains when reference-program execution failures are controlled for?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Hcye

Thank you for pointing out that Hard changes both the input and the scoring rule. We hope the comparisons below address your questions about grounding and reference execution. We also revised the presentation throughout; the numbers below refer to the revised PDF.

(a) Decision errors and grounding failures

Across the 140 primary configurations, the Hard–Base gap was −25.9 points under strict scoring and −6.3 when we relaxed the grounding check on the same outputs (Table 4). Table 61 gives a direct comparison on ten Palimpzest–Qwen3 Hard configurations. The same responses passed the decision checks in 73.1% of cases and the full contract in 42.6%. Most of the headline drop thus comes from the grounding contract rather than from wrong decisions.

Of 34,665 observed Hard rejections after excluding infrastructure failures, 55.7% had a grounding flag without a decision flag, 18.3% a decision flag without a grounding flag, 22.7% both, and 3.3% neither (Table 51). Other checks can fail on the same record.

An LLM judge scored all 81,189 complete Base–Hard pairs. Three database graduate students independently rated each of 400 sampled outputs from both conditions, including judge–verifier disagreements. They did not see the condition label or either automated verdict. We used the majority rating for each output, weighted the groups within each configuration and averaged across 140 configurations. The Hard–Base gap was −20.1 points [−26.4, −13.6] for broad validity and −1.8 [−8.5, +4.6] for decision correctness. The former decline exceeded the latter by 18.3 points [11.6, 25.3] (Table 5; Appendix G.1). We now report the three scorings and the human estimates together in the abstract and Table 4, with their populations in the introduction and the Table 4 caption.

(b) Translation after matching reference success

On the 63 template–anchor pairs where both executors accepted the reference, step-by-step requests yielded higher program acceptance than goal-only requests. The difference was 10.7 points on Qwen3 (95% CI [1.0, 21.0]) and 18.7 on GPT-5.4-mini [7.4, 32.2]. Across all nonempty targets, the differences were 4.9 [1.5, 8.8] and 6.8 [2.1, 12.4], respectively (Table 3). Reference acceptance was 22.0% on Qwen3 and 46.7% on GPT-5.4-mini. The Step advantage remains on the shared subset, where 52 of 63 cases use one or two operators. When reference failures are controlled, the trend thus remains and grows: on this subset, 18.3% of goal-only executions fail with program errors under both executors, against none of the step-by-step executions, and Step still exceeds Goal by 11.5 points [4.2, 21.3] when both executors must accept (Appendix E.2).

Thank you for your responses. These responses have addressed my concerns. I will keep my original rating.

Reviewer_A5uxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

SemBench-Hard evaluates LLM-backed semantic query processing engines by hardening translation through goal-only queries and execution through distractors constraints output schemas and strict grounding checks across four datasets. It demonstrates that standard evaluations overstate reliability because most failures stem from missing or incorrect grounding rather than wrong decisions.

Critical strengths
  1. Addresses a critical blind spot in current SQPE and agent benchmarks by decoupling and stressing both query translation and operator-level grounding.

  2. Extensive evaluation across multiple datasets, operators, backbones, and system adapters, supported by rigorous human calibration.

Critical weaknesses
  1. The "Hard" tasks rely heavily on heuristic or rule-based dataset perturbations, which may not fully reflect the diversity of real-world ambiguity and noisy user inputs.

  2. Limited effectiveness of bounded tools in closing the performance gap suggests that current local tool designs are still insufficient for complex grounding recovery.

Questions for authors
  1. How do the insights of SemBench-Hard generalize to hybrid semantic-relational query environments that handle multi-modal or unstructured tables differently?

  2. The authors might consider citing and discussing the related work CoreSemDB: Benchmarking Hybrid Semantic-Relational Query Processing over Text-Rich Databases in their manuscript.

2 comments
AuthorsOpenReview ↗

Response to Reviewer A5ux

Thank you for your constructive comments and for pointing us to CoreSemDB. We hope the noise and tool comparisons, the clarification of our scope, and the related-work revision below address all of your concerns. We also revised the presentation throughout and highlighted the changes in blue. The section, table, and appendix numbers below refer to the revised PDF.

W1. Ambiguity and noisy requests

We tested the two parts of this concern that we can measure: noisy inputs, and distractors that our rules did not select. To test noisy requests, we rewrote the requests informally, with typos, slang, and dropped punctuation. To test noisy records, we added typos, casing errors, and stray punctuation to the records. Both tests preserved the labels and evidence identifiers. On 4,779 Palimpzest–Qwen3 Hard tasks, request noise changed acceptance by 0.0 points [−1.1, +1.2] and record noise by +0.3 [−0.8, +1.5] (Appendix I.3). To test distractors that our rules did not select, we used HotpotQA's own. Across 12 Qwen3 configurations, the Base-to-Hard drop was 17.2 points under strict scoring and 2.7 points when the grounding check was relaxed (Appendix I.5). These tests cover surface noise and distractors, not ambiguous user intent. Estimating how often ambiguous intent causes failures would require a sample of deployed requests. We now make that scope explicit in §2.4.

W2. Can bounded tools recover the gap?

They recover part of the gap. The tools are deterministic lookup tools that see only the task's input, with up to three calls per operator. After we restored the missing WDC and ContractNLI inputs, tools recovered 7.9 of the 31.3 points between unaided Hard and Base across the 21 affected Qwen3 configurations (Appendix J.2). In a separate Palimpzest–Qwen3 comparison outside SWE-bench MAP, generic search/open/Python tools changed acceptance by −3.9 points [−5.2, −2.6] and task-specific tools by +4.1 [2.5, 5.6], both relative to unaided Hard (Appendix J).

We then examined why failures remain. To see whether the tools retrieved the grounding, we took the failed Hard+tool records where a tool could return it. In 60% of them, the tool outputs already contained it (94% in WDC Products). To test the call budget, we reran the 315 SWE-bench RANK tasks with three calls instead of eight. Qwen3 Hard+tool acceptance changed by −0.6 points [−3.4, +2.1] (Appendix J.3). In SWE-bench MAP, a helper tool proposes the files to edit from the shown code snippets. The helper alone passed 402 of 500 tasks because it reads the snippet source labels. With the labels removed and the snippets shuffled, it passed 196, and Qwen3's tool recovery fell from +45.3 to +5.9 points (Appendix J.1, Table 74). We agree that current tool designs do not solve the use of the grounding that tools return. In §4.4 and Figure 5a, we now separate the nine SWE-bench MAP configurations from the other 106 three-call configurations.

Q1. Hybrid plans, table representations, and other modalities

We tested text-rich records in a fixed relational representation. Two points bear on other environments. The closest evidence is Nirvana, a multimodal analytics engine: on the same text FILTER tasks as our adapters, its operators were accepted on 90.5% of Base tasks and 61.5% of Hard tasks (Claude Sonnet 4.6; Table 67a). The second point is where our two failures occur. Both concern what one step hands to the next: the fields that a program reads, and the grounding that an operator returns with its decision. A hybrid plan has the same hand-off when an SQL aggregate consumes a semantic extraction, and a hybrid benchmark could check the extraction at that point. Our experiments did not vary the table representation, execute mixed SQL–semantic aggregation, or use native image or audio inputs (Appendix K).

Q2. CoreSemDB

We now discuss Tian et al. (COLM 2026) in the introduction, in §5, and in Appendix B. CoreSemDB evaluates hybrid semantic-relational queries over text-rich databases. We execute generated programs to score translation and use fixed reference programs to test whether each operator's output satisfies its contract.

Thank you very much for your detailed reply. I will maintain my original rating.

Reviewer_JWmDOpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
1(−1)
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SemBench-Hard to evaluate semantic query systems along two axes: translating natural language into operator programs, and producing correct decisions with appropriate grounding during execution. It compares Base and Hard tasks and examines human calibration, operator chains, and local tool use.

Critical strengths
  1. Separating translation from execution and disclosing evaluation limitations.

    This decomposition helps locate interface and grounding failures. The appendices disclose implementation details and sensitivity analyses that qualify the conclusions and provide concrete points for verification.

  2. Distinguishing decision correctness from grounding quality.

    The evaluation distinguishes correct decisions from outputs that also satisfy the evidence requirements. Comparing strict and relaxed scoring helps identify how much of the acceptance gap comes from grounding checks, providing more diagnostic information than aggregate accuracy alone.

Critical weaknesses
  1. The human calibration is incomplete, limiting the interpretation of the main gap as a decline in semantic usability.

    The acceptance gap changes from −25.9 percentage points under strict scoring to −6.3 when grounding requirements are relaxed, showing substantial dependence on the evidence-scoring rules. Appendix F mainly adjusts Hard outcomes, without applying the same human standard to Base or covering all rejection categories. Table 27 also shows appreciable human validity among rejections outside the main calibration. The calibrated gaps therefore do not directly establish the difference in semantic usability.

  2. The translation and tool analyses do not sufficiently separate execution configurations from model capabilities.

    Reference-program acceptance is only 22.0%, rising to 46.7% with the alternative executor. Under Qwen3, 55 of 67 accepted reference executions come from short programs, so conditional scoring favors the subset the executor can handle. For tools, the SWE MAP helper alone achieves 80.4% acceptance, and its nine cells contribute 3.0 percentage points of the overall 4.0-point recovery. These results do not sufficiently distinguish translation quality, executor limitations, and deterministic helper contributions.

Questions for authors
  1. Can you provide symmetric human calibration and a paired translation comparison?

    Please apply consistent human criteria to the main acceptance and rejection categories in both Base and Hard, and report paired Step–Goal differences with confidence intervals on the subset where both executors accept the reference program.

  2. Do the specialized tools receive the necessary inputs?

    In the released code, _build_hard_tool_task_view omits WDC offer fields and ContractNLI clause and hypothesis fields, causing the corresponding tools to return missing-field indicators or empty results. Please confirm the version used for the experiments and provide tool inputs, outputs, and recovery results before and after correcting this issue.

  3. How do supervision fields and post-processing affect the tool results?

    Nested offer records can expose gold_label and cluster_id through retrieval, while SWE MAP post-processing replaces model-predicted files with helper-proposed files. Please report results after removing supervision fields and distinguish the original model predictions, helper-only outputs, and final scored outputs.

9 comments
AuthorsOpenReview ↗

Response to Reviewer JWmD (1/3)

Thank you for checking our code and identifying the missing inputs to the WDC and ContractNLI tools. We hope the calibration, matched translation comparison, and corrected tool results below address all of your concerns.

W1 / Q1(a). Symmetric human calibration

Following your suggestion, three database graduate students independently rated each of the 400 sampled Base and Hard outputs under the same rubric. They checked decision correctness, completeness, and whether the displayed sources supported the required evidence. We used the majority of the three ratings for each output. Under this single standard, the Hard–Base gap is −20.1 points, closer to strict acceptance (−26.0) than to grounding-relaxed acceptance (−6.3), while decisions differ by only −1.8.

GPT-6.1 Sol scored all 162,378 outputs from 81,189 complete Base–Hard pairs. We grouped the outputs by condition, judge verdict, and verifier verdict. The human sample comprised both outputs for 100 randomly selected anchors and 50 additional outputs from each of four judge–verifier disagreement groups. The students saw the task and sources. The condition label and both automated verdicts were hidden. We weighted each group's human validity rate by its share of outputs within each configuration and averaged the 140 configuration estimates.

Measure on complete pairsBaseHardHard − Base, pp [95% CI]
Verifier acceptance87.1%61.1%−26.0
Broad validity, LLM judge alone75.3%53.8%−21.5 [−22.0, −20.9]
Broad validity, human-calibrated79.7%59.6%−20.1 [−26.4, −13.6]
Decision correct, human-calibrated85.4%83.6%−1.8 [−8.5, +4.6]

Among verifier rejections, the estimated share of broadly valid outputs was 28.7% on Base and 34.5% on Hard. The decline in broad validity exceeded the decline in decision correctness by 18.3 points [11.6, 25.3]. Weighting outputs instead of configurations gave a broad-validity gap of −19.3 points. Bounding the 1.2% of outputs marked uncertain by the judge gave a range of −21.0 to −19.3. With 120 additional outputs in the six sparsest groups, each rated by the same three students, the broad-validity gap was −22.3 [−28.3, −16.1] and the decision gap was −1.0 [−5.2, +3.5].

The new panel also re-rated 87 previously assessed Hard outputs that the verifier had accepted, without seeing the earlier labels. It judged 69.0% broadly valid. Table 28 (Table 27 in the submitted version) reported 43.8% for its full set of 89 accepted outputs. The difference reflects how the two panels classified borderline outputs: the new panel judged more of the same outputs wrong or insufficiently grounded (16.1% against 10.3%). Appendix F.3 (Tables 30 and 31) reports the population size, human sample size, and valid count by group and rejection category.

AuthorsOpenReview ↗

Response to Reviewer JWmD (2/3)

W2 / Q1(b). Step versus Goal on jointly executable references

Both executors accepted the reference on 63 template–anchor pairs from 19 templates. Across four translators (252 comparisons per executor), programs generated from step-by-step requests were accepted more often than those generated from goal-only requests. The differences were 10.7 points on Qwen3 [1.0, 21.0] and 18.7 on GPT-5.4-mini [7.4, 32.2]. The 95% intervals resample scenario–template clusters. On all nonempty targets, reference acceptance was 22.0% and 46.7% and the Step gains were 4.9 and 6.8 points, respectively. Only 11 common cases had three to five operators (§4.1). Therefore, the goal-only drop remains when execution is restricted to references both executors run correctly, which separates translation quality from executor limitations.

The all-target gains do not condition on reference acceptance, so they do not depend on which references the executor can run. On the common subset, the two executors agreed on 81.0% of generated-program verdicts, and we attribute to the translator only failures that both record identically. Of the 252 goal-only executions, 46 (18.3%) failed with program errors under both executors, against none of the step-by-step executions. When success requires both executors to accept, Step exceeds Goal by 11.5 points [4.2, 21.3].

Q2. Missing helper inputs

In the submitted Qwen3 runs, we omitted the WDC offer fields and the ContractNLI clause and hypothesis from the helper inputs. This understated the benefit of the tools on the 21 affected configurations. Restoring the inputs raised the gain from 3.8 to 7.9 points (+4.1 points, 95% CI [3.3, 5.0]). Corrected tool-assisted acceptance was 48.3%, still 23.4 points below Base (71.7%). The main comparison in Table 4 used no tools. Therefore, the omission understated tool recovery but does not affect the main Base–Hard comparison, and the corrected tools still leave most of the gap.

21 Qwen3 configurationsHard, no toolsSubmitted toolsFixed toolsBase
Acceptance40.4%44.2%48.3%71.7%

We kept the task keys, prompts, call budget, and verifier fixed. Nonempty returns rose from 0/17,108 calls to 16,129/16,129. The model chose whether to retry. On one WDC JOIN task, both titles and prices were missing before the fix. Afterwards, the helper returned a title overlap of 0.78 and prices of 39.90 EUR and 39.90 EUR, and the verdict changed from rejected to accepted. The helper applies the benchmark's attribute rule and agrees with it on all 7,155 calls that carry two prices.

Submitted code: 88574983. Corrected code: 808db118. Paired offline helper input/output traces for the same tasks: WDC JOIN pair:416796#56475885 and ContractNLI JOIN contract:34/hypothesis:nda-4/span:19 (experiments/checks/tool_reruns/helper_traces.json).

AuthorsOpenReview ↗

Response to Reviewer JWmD (3/3)

W2 / Q3. Supervision fields and post-processing

In six Claude WDC configurations, nested retrieval results exposed gold_label or cluster_id to the model. We removed these and all other supervision fields. Exposure fell from 55–58/100 records per JOIN cell and 30/30 per RANK cell to zero. Acceptance changed from 46.1% to 49.6% (+3.5 points, 95% CI [−3.1, +10.5]). The interval includes both a decrease and an increase. Across all tool outputs returned to the model, exposure occurs only in these six configurations (261 of 64,518 Hard+tool records) and in none of the 21,609 rerun records. Recomputing Table 41 (Table 37 in the submitted version) with the missing inputs restored and the exposed fields removed gives +2.0 points [1.1, 2.9] in the 106 three-call configurations outside SWE-bench MAP (submitted: +1.0). Therefore, the exposed fields did not drive the tool results, and with both corrections, tools recover only 2.0 points outside SWE-bench MAP.

The nine SWE-bench MAP cells contributed 3.0 of the 4.0 points of recovery in submitted Table 37. We rescored each stage.

SWE-bench MAPNo toolsModel after helper¹Helper aloneFile overwrite off²Test selection off²Final
Qwen3 (3 × 500)39.9%83.1%80.4%85.0%84.1%85.1%
GPT-5.4-mini (3 × 500)34.3%73.7%80.4%84.1%75.5%85.7%
Sonnet 4.6 (3 × 100)47.3%55.7%75.0%73.3%57.0%73.3%

¹ Model output before post-processing. ² Other post-processing enabled.

File replacement adds 0.0–1.6 points. All post-processing together adds 2.0, 12.0, and 17.6 points, respectively. Test selection, which picks one test among the model's, the helper's, and the visible failing tests, adds 0.9, 10.2, and 16.3 points. Per task, the final output is correct where the helper alone fails on 75, 146, and 48 tasks, and wrong where the helper succeeds on 5, 67, and 53 (net +4.7, +5.3, and −1.7 points). Therefore, the SWE-bench MAP recovery comes from the helper and post-processing, and the model adds little beyond the helper.

The SWE-bench MAP context also labels the gold-derived snippet by its source and shows it first. We reran Qwen3 on all 500 tasks per system without the labels, and with the order also shuffled, rerunning the labeled condition on the same host as a control. Hard acceptance stays at 39.9%, 39.6%, and 39.1%. Tool recovery falls from +45.3 to +30.1 and +5.9 points [0.3, 11.4], and the helper alone from 80.4% to 62.8% and 39.2%. Therefore, the Base–Hard gap does not depend on this cue, while most of the SWE-bench MAP recovery does. We now report these nine cells separately from the other 106 three-call cells.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.