- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SemBench-Hard, a benchmark for natural-language query translation and the execution of LLM-backed semantic operators. It evaluates whether operators return not just correct decisions but also the evidence or structured outputs needed by downstream steps. Across multiple datasets, systems, and models, the authors find that grounding requirements reveal substantial failures that decision-only evaluation would miss.
Critical strengths
The paper addresses a practical and underexplored problem. I like the separation between query translation and operator execution, and especially the focus on whether intermediate outputs are actually usable. The experiments cover a good range of settings, and the human evaluation and ablations help substantiate the main observation rather than relying on one benchmark score.
Critical weaknesses
My main concern is how the Base–Hard gap is interpreted. Hard tasks change both the input and the success criteria, so the headline drop does not isolate a decline in semantic reasoning. The relaxed and human-calibrated grounding results deserve at least as much emphasis as strict acceptance. The translation results are also harder to interpret because the reference programs themselves have fairly low success rates under the primary executor.
Questions for authors
Could the authors more clearly separate decision errors from failures of the grounding contract? Also, how much of the translation trend remains when reference-program execution failures are controlled for?
2 comments
Thank you for your responses. These responses have addressed my concerns. I will keep my original rating.
Response to Reviewer Hcye
Thank you for pointing out that Hard changes both the input and the scoring rule. We hope the comparisons below address your questions about grounding and reference execution. We also revised the presentation throughout; the numbers below refer to the revised PDF.
(a) Decision errors and grounding failures
Across the 140 primary configurations, the Hard–Base gap was −25.9 points under strict scoring and −6.3 when we relaxed the grounding check on the same outputs (Table 4). Table 61 gives a direct comparison on ten Palimpzest–Qwen3 Hard configurations. The same responses passed the decision checks in 73.1% of cases and the full contract in 42.6%. Most of the headline drop thus comes from the grounding contract rather than from wrong decisions.
Of 34,665 observed Hard rejections after excluding infrastructure failures, 55.7% had a grounding flag without a decision flag, 18.3% a decision flag without a grounding flag, 22.7% both, and 3.3% neither (Table 51). Other checks can fail on the same record.
An LLM judge scored all 81,189 complete Base–Hard pairs. Three database graduate students independently rated each of 400 sampled outputs from both conditions, including judge–verifier disagreements. They did not see the condition label or either automated verdict. We used the majority rating for each output, weighted the groups within each configuration and averaged across 140 configurations. The Hard–Base gap was −20.1 points [−26.4, −13.6] for broad validity and −1.8 [−8.5, +4.6] for decision correctness. The former decline exceeded the latter by 18.3 points [11.6, 25.3] (Table 5; Appendix G.1). We now report the three scorings and the human estimates together in the abstract and Table 4, with their populations in the introduction and the Table 4 caption.
(b) Translation after matching reference success
On the 63 template–anchor pairs where both executors accepted the reference, step-by-step requests yielded higher program acceptance than goal-only requests. The difference was 10.7 points on Qwen3 (95% CI [1.0, 21.0]) and 18.7 on GPT-5.4-mini [7.4, 32.2]. Across all nonempty targets, the differences were 4.9 [1.5, 8.8] and 6.8 [2.1, 12.4], respectively (Table 3). Reference acceptance was 22.0% on Qwen3 and 46.7% on GPT-5.4-mini. The Step advantage remains on the shared subset, where 52 of 63 cases use one or two operators. When reference failures are controlled, the trend thus remains and grows: on this subset, 18.3% of goal-only executions fail with program errors under both executors, against none of the step-by-step executions, and Step still exceeds Goal by 11.5 points [4.2, 21.3] when both executors must accept (Appendix E.2).