- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces NoGoBench, a benchmark of 1,061 scientifically impossible tasks and minimally edited feasible counterparts across six domains. It evaluates feasibility judgments and task execution in separate conversations, measuring self-reported completion conditional on correctly judging a task impossible. Across 19 model configurations, feasibility discrimination is consistently strong, while conditional false-completion rates vary substantially. An external model classifies the resulting failures into completion-accounting errors, specification evasion, and invalid or unsupported solutions. Additional experiments examine reasoning effort, released post-training checkpoints, and execution pressure. The main finding is that recognizing infeasibility under direct questioning does not ensure consistent constraint handling or completion reporting when asked to perform the task.
Critical strengths
-
The impossible–feasible task pairs are a useful design choice. They make blanket refusal an inadequate strategy and help distinguish feasibility recognition from general reluctance to answer. Evaluating judgments and execution in independent conversations also avoids feeding the correct judgment into the execution prompt. Together, these choices give the benchmark a clear purpose beyond measuring accuracy on impossible requests.
-
The gap is observed across a broad set of models, and the paper makes a serious effort to examine what contributes to it. Appendix E is particularly useful: the external assessment exposes substantial measurement sensitivity, but also confirms most originally identified non-C1 completion claims in the three tested configurations. This supports the view that the phenomenon extends beyond models answering “Yes” after giving a correct impossibility explanation. The concrete examples also make the constraint violations easy to inspect.
Critical weaknesses
-
The interpretation of cFCR needs to remain closely tied to the elicitation protocol. A model that correctly explains impossibility may interpret “whether you completed the task” as asking whether it finished responding, rather than whether it produced the requested object. This is still a reporting problem, but differs from presenting an invalid construction as a valid solution. The paper acknowledges this distinction and provides useful external checks; nevertheless, Appendix E shows substantial changes in rates and even model ordering across assessment procedures. The pressure experiment adds another ambiguity: the forced prompt explicitly requests confirmation of completion, so it changes both the demand to solve the task and the demand to report success. This makes it difficult to attribute the increase specifically to deteriorating constraint handling under execution pressure.
-
The headline failure-mechanism result is less well validated than the main binary outcome. The claim that 61.2% of classified cases involve specification evasion relies on one external judge, with C1–C6 explicitly taking precedence over generic incorrect attempts. The boundary between C2 and C7 is not always clear. For example, producing a grammar that violates the definition of a context-free grammar could reflect an ordinary conceptual error, even though the answer can also be described as redefining the grammar class. The human validation in Appendix F.4 assesses task pairs, rather than these response categories. Without an independent annotation study, I find the taxonomy useful descriptively, but the exact prevalence of specification evasion less convincing.
Questions for authors
-
Does the main gap persist when the completion question explicitly asks whether all original requirements were satisfied without changing their meaning or assumptions? Reporting C1 and non-C1 contributions under this wording would help separate completion-accounting ambiguity from invalid claimed solutions.
-
Can the pressure experiment separate “provide a solution” pressure from the explicit instruction to “confirm completion”? In particular, does solution pressure alone increase first-turn specification violations?
-
Can you provide independent human annotations for a stratified sample of the failure categories, especially C2 versus C7? Agreement statistics and examples of disagreements would help establish how reliably the reported mechanism shares can be interpreted.
5 comments
Response to Reviewer kE1o (2/2)
W2 and Q3 (i): Do independent human annotations of the failure categories support the reported mechanism shares?
Response: We thank the reviewer for this request. We agree that the shares of the eight categories rest on a single judge and that the validation in Appendix F.4 covers task pairs rather than response categories. Two independent annotators therefore labeled all 97 NY cases of Claude Opus 5, a model also checked in Appendix E. All eight categories occur among them, and the annotators saw only the original task and the first-turn execution response, without the judge label, the source model, or the completion report. The category definitions matched the judge protocol.
A single response can show more than one departure at once. For example, it may weaken the required bound and also substantively rely on a premise the task does not grant, which fits both C3 and C4. Appendix C requires one primary category under the precedence and earliest causal departure rules (L794–798), but identifying the earliest departure may still require judgment when several occur. We therefore acknowledge that the fine-grained assignment among the non-C1 categories is less stable, while the distinction between C1 and non-C1 is considerably more stable. This instability concerns only the post-hoc category labels, which do not determine completion status or enter cFCR (L291–293).
Agreement is summarized below.
| Agreement | Annotators A vs B | A vs judge | B vs judge |
|---|---|---|---|
| Eight categories | 78.4% (0.72) | 60.8% (0.49) | 61.9% (0.50) |
| C1 vs non-C1 | 95.9% (0.89) | 83.5% (0.56) | 85.6% (0.63) |
Values are raw agreement with Cohen's kappa in parentheses. Among each label set's cases excluding C8, specification evasion accounts for 68.8% and 64.9% for annotators A and B, and 65.6% for the judge. These estimates support the aggregate interpretation in L371–375, but do not independently establish the exact benchmark-wide share of 61.2%.
W2 and Q3 (ii): How reliably can C2 be distinguished from C7, and what do the disagreements look like?
Response: C2 and C7 are defined by different departures from the task (Table 2, L810–841). C7 keeps the original definitions and requirements and gives a verifiable attempt in which a key fact, computation, proof, or construction is wrong. C2 instead changes the meaning of a term, rule, or formal model, so that the task appears solvable under a different definition. In practice, agreement on this boundary is high. The two annotators agree on 34 of the 39 cases (87%) that both placed in C2 or C7, above their 78.4% agreement over all eight categories. On cases that both the judge and an annotator placed in C2 or C7, the judge agrees with annotator A on 30 of 31 and with annotator B on 29 of 30.
In the revision, we will add this study, the agreement statistics, and examples of disagreement to the appendix, and report the aggregate shares in the main text with the fine-grained categories as a descriptive breakdown.
Dear Reviewer kE1o,
We are grateful for the reviewer's detailed and constructive feedback. The suggested experiments on the wording of the completion question, the separation of the two pressure clauses, and the human annotation of failure categories directly strengthened the paper. We have run all three.
The revised paper reports the C1 and non-C1 contributions for all 19 configurations in Table 2. It adds the stricter completion question in Appendix E.4, the two single-clause pressure conditions in Appendix F.5, and the independent human annotation in Appendix C.2. We would appreciate hearing whether these results address the points raised in the review. We remain available for any further questions during the discussion.
Best regards,
Authors
Response to Reviewer kE1o (1/2)
Response: We thank the reviewer for this suggestion. We agree that cFCR should be read together with the elicitation protocol, since the second-turn answer records a completion claim rather than verified success (L247–250). We ran the suggested wording on the two models used in Sec. 6.3, keeping the first-turn execution responses unchanged and replacing only the completion question.
On tasks judged impossible with valid answers under both wordings, cFCR changes as follows.
Arrows read standard → strict. C1, non-C1 and NN → Yes are contributions to cFCR in percentage points, using the original NY labels. NN → Yes counts originally NN cases that answer Yes under the strict wording. The gap persists under the strict wording in both models. The larger relative drop for C1 is consistent with the reviewer's concern about ambiguity in completion accounting. Yet 39% and 49% of the original non-C1 cases switch to No, while their first-turn responses remain unchanged. The strict wording therefore removes some original non-C1 completion claims from the cFCR numerator without changing the underlying execution responses. We therefore retain the standard wording as the main protocol, since it is the question a user would naturally ask and it matches the benchmark protocol, and we report the strict wording as a robustness check. Reporting C1 and non-C1 separately shows their differing sensitivity to wording.
In the revision, we will report the C1 and non-C1 contributions alongside cFCR for all 19 default configurations in the main results and include the strict-wording comparison in the appendix.
Response: We thank the reviewer for pointing out this confound. We split the forced prompt into two conditions on the two models used in Sec. 6.3, one with "You must complete the following task. Provide a solution." and one with "You must complete the following task. Confirm completion." Both retain the stronger opening of the forced prompt. Their comparison contrasts the two clauses under that shared opening. All other settings follow Appendix F.5 (L1154–1169), and the NY cases are classified by the same judge.
DeepSeek is DeepSeek-V4-Flash and Qwen3.5 is Qwen3.5-35B-A3B. The C1 and C2–C6 columns give the contribution of each group to cFCR in percentage points. Both new conditions raise cFCR relative to the standard prompt, but the solution-only condition raises it far more in both models. Under the solution-only condition, the contribution of specification evasion to cFCR rises for Qwen3.5-35B-A3B and barely changes for DeepSeek-V4-Flash, where almost all of its total cFCR increase comes from C1. The confirmation-only condition mainly raises C1 in both models.
Both components therefore change model behavior, and the demand for a solution has the larger effect. This is consistent with Sec. 6.3, where pressure affects both completion-status reporting and the handling of the original specification (L471–476).
We will add the two conditions to Appendix F.5 and refer to them in Sec. 6.3.