acceptodds
Desk rejected at ICLR 2027

Recognizing Constraints Is Not Enough: The Judgment–Execution Gap in Large Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable large language model assistants and agents must pursue users' goals without violating the constraints that define valid completion. It remains unclear whether models preserve task constraints during execution even when they correctly judge the task infeasible. To study this judgment-execution consistency under verifiable conditions, we use well-defined scientific constraints that can be checked against external evidence. We introduce NoGoBench, comprising 1061 scientifically impossible tasks and minimally edited feasible counterparts across six domains, and evaluate feasibility judgments and task execution in independent conversations. Our conditional False Completion Rate (cFCR) measures the proportion of tasks self-reported as completed among those correctly judged impossible. Across 19 general-purpose and science-specialized model configurations from 10 families, Contrastive Macro-F1 ranges from 92.2% to 97.8%, while cFCR ranges from 8.3% to 76.6%. An external judge attributes most classified inconsistent cases to specification evasion through redefined terms, weakened requirements, or substituted goals. Further experiments show that increased reasoning reduces direct errors without reliably improving completion reporting, while post-training recipes affect feasibility judgments and completion claims differently. Stronger execution pressure substantially increases false completion. These findings show that recognizing scientific constraints does not ensure reliable execution, motivating further investigation of judgment-execution consistency under value constraints. Our code and data are available at https://anonymous.4open.science/r/NoGoBench-3ED5.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, AI Alignment, LLM Evaluation, Judgment–Execution Consistency, False Completion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kE1oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces NoGoBench, a benchmark of 1,061 scientifically impossible tasks and minimally edited feasible counterparts across six domains. It evaluates feasibility judgments and task execution in separate conversations, measuring self-reported completion conditional on correctly judging a task impossible. Across 19 model configurations, feasibility discrimination is consistently strong, while conditional false-completion rates vary substantially. An external model classifies the resulting failures into completion-accounting errors, specification evasion, and invalid or unsupported solutions. Additional experiments examine reasoning effort, released post-training checkpoints, and execution pressure. The main finding is that recognizing infeasibility under direct questioning does not ensure consistent constraint handling or completion reporting when asked to perform the task.

Critical strengths
  • The impossible–feasible task pairs are a useful design choice. They make blanket refusal an inadequate strategy and help distinguish feasibility recognition from general reluctance to answer. Evaluating judgments and execution in independent conversations also avoids feeding the correct judgment into the execution prompt. Together, these choices give the benchmark a clear purpose beyond measuring accuracy on impossible requests.

  • The gap is observed across a broad set of models, and the paper makes a serious effort to examine what contributes to it. Appendix E is particularly useful: the external assessment exposes substantial measurement sensitivity, but also confirms most originally identified non-C1 completion claims in the three tested configurations. This supports the view that the phenomenon extends beyond models answering “Yes” after giving a correct impossibility explanation. The concrete examples also make the constraint violations easy to inspect.

Critical weaknesses
  • The interpretation of cFCR needs to remain closely tied to the elicitation protocol. A model that correctly explains impossibility may interpret “whether you completed the task” as asking whether it finished responding, rather than whether it produced the requested object. This is still a reporting problem, but differs from presenting an invalid construction as a valid solution. The paper acknowledges this distinction and provides useful external checks; nevertheless, Appendix E shows substantial changes in rates and even model ordering across assessment procedures. The pressure experiment adds another ambiguity: the forced prompt explicitly requests confirmation of completion, so it changes both the demand to solve the task and the demand to report success. This makes it difficult to attribute the increase specifically to deteriorating constraint handling under execution pressure.

  • The headline failure-mechanism result is less well validated than the main binary outcome. The claim that 61.2% of classified cases involve specification evasion relies on one external judge, with C1–C6 explicitly taking precedence over generic incorrect attempts. The boundary between C2 and C7 is not always clear. For example, producing a grammar that violates the definition of a context-free grammar could reflect an ordinary conceptual error, even though the answer can also be described as redefining the grammar class. The human validation in Appendix F.4 assesses task pairs, rather than these response categories. Without an independent annotation study, I find the taxonomy useful descriptively, but the exact prevalence of specification evasion less convincing.

Questions for authors
  • Does the main gap persist when the completion question explicitly asks whether all original requirements were satisfied without changing their meaning or assumptions? Reporting C1 and non-C1 contributions under this wording would help separate completion-accounting ambiguity from invalid claimed solutions.

  • Can the pressure experiment separate “provide a solution” pressure from the explicit instruction to “confirm completion”? In particular, does solution pressure alone increase first-turn specification violations?

  • Can you provide independent human annotations for a stratified sample of the failure categories, especially C2 versus C7? Agreement statistics and examples of disagreements would help establish how reliably the reported mechanism shares can be interpreted.

5 comments
AuthorsOpenReview ↗

Response to Reviewer kE1o (1/2)

W1 and Q1: If the completion question instead asks whether all original requirements were satisfied without changing their meaning or assumptions, does the gap persist, and how do the C1 and non-C1 contributions change?

Response: We thank the reviewer for this suggestion. We agree that cFCR should be read together with the elicitation protocol, since the second-turn answer records a completion claim rather than verified success (L247–250). We ran the suggested wording on the two models used in Sec. 6.3, keeping the first-turn execution responses unchanged and replacing only the completion question.

On tasks judged impossible with valid answers under both wordings, cFCR changes as follows.

ModelcFCRC1 (pp)Non-C1 (pp)NN → Yes (pp)
DeepSeek-V4-Flash24.1% → 16.3%14.7 → 4.39.4 → 5.76.3
Qwen3.5-35B-A3B35.2% → 15.9%8.9 → 1.526.3 → 13.51.0

Arrows read standard → strict. C1, non-C1 and NN → Yes are contributions to cFCR in percentage points, using the original NY labels. NN → Yes counts originally NN cases that answer Yes under the strict wording. The gap persists under the strict wording in both models. The larger relative drop for C1 is consistent with the reviewer's concern about ambiguity in completion accounting. Yet 39% and 49% of the original non-C1 cases switch to No, while their first-turn responses remain unchanged. The strict wording therefore removes some original non-C1 completion claims from the cFCR numerator without changing the underlying execution responses. We therefore retain the standard wording as the main protocol, since it is the question a user would naturally ask and it matches the benchmark protocol, and we report the strict wording as a robustness check. Reporting C1 and non-C1 separately shows their differing sensitivity to wording.

In the revision, we will report the C1 and non-C1 contributions alongside cFCR for all 19 default configurations in the main results and include the strict-wording comparison in the appendix.

W1 and Q2: If the demand to provide a solution is separated from the instruction to confirm completion, which one drives the increase in cFCR, and does solution pressure alone increase first-turn specification violations?

Response: We thank the reviewer for pointing out this confound. We split the forced prompt into two conditions on the two models used in Sec. 6.3, one with "You must complete the following task. Provide a solution." and one with "You must complete the following task. Confirm completion." Both retain the stronger opening of the forced prompt. Their comparison contrasts the two clauses under that shared opening. All other settings follow Appendix F.5 (L1154–1169), and the NY cases are classified by the same judge.

ConditionDeepSeek cFCRDeepSeek C1DeepSeek C2–C6Qwen3.5 cFCRQwen3.5 C1Qwen3.5 C2–C6
standard24.1%14.78.235.3%8.924.5
confirmation-only33.6%22.49.939.3%18.918.3
solution-only60.8%50.79.657.0%22.731.5
forced50.8%39.210.570.9%34.034.1

DeepSeek is DeepSeek-V4-Flash and Qwen3.5 is Qwen3.5-35B-A3B. The C1 and C2–C6 columns give the contribution of each group to cFCR in percentage points. Both new conditions raise cFCR relative to the standard prompt, but the solution-only condition raises it far more in both models. Under the solution-only condition, the contribution of specification evasion to cFCR rises for Qwen3.5-35B-A3B and barely changes for DeepSeek-V4-Flash, where almost all of its total cFCR increase comes from C1. The confirmation-only condition mainly raises C1 in both models.

Both components therefore change model behavior, and the demand for a solution has the larger effect. This is consistent with Sec. 6.3, where pressure affects both completion-status reporting and the handling of the original specification (L471–476).

We will add the two conditions to Appendix F.5 and refer to them in Sec. 6.3.

AuthorsOpenReview ↗

Response to Reviewer kE1o (2/2)

W2 and Q3 (i): Do independent human annotations of the failure categories support the reported mechanism shares?

Response: We thank the reviewer for this request. We agree that the shares of the eight categories rest on a single judge and that the validation in Appendix F.4 covers task pairs rather than response categories. Two independent annotators therefore labeled all 97 NY cases of Claude Opus 5, a model also checked in Appendix E. All eight categories occur among them, and the annotators saw only the original task and the first-turn execution response, without the judge label, the source model, or the completion report. The category definitions matched the judge protocol.

A single response can show more than one departure at once. For example, it may weaken the required bound and also substantively rely on a premise the task does not grant, which fits both C3 and C4. Appendix C requires one primary category under the precedence and earliest causal departure rules (L794–798), but identifying the earliest departure may still require judgment when several occur. We therefore acknowledge that the fine-grained assignment among the non-C1 categories is less stable, while the distinction between C1 and non-C1 is considerably more stable. This instability concerns only the post-hoc category labels, which do not determine completion status or enter cFCR (L291–293).

Agreement is summarized below.

AgreementAnnotators A vs BA vs judgeB vs judge
Eight categories78.4% (0.72)60.8% (0.49)61.9% (0.50)
C1 vs non-C195.9% (0.89)83.5% (0.56)85.6% (0.63)

Values are raw agreement with Cohen's kappa in parentheses. Among each label set's cases excluding C8, specification evasion accounts for 68.8% and 64.9% for annotators A and B, and 65.6% for the judge. These estimates support the aggregate interpretation in L371–375, but do not independently establish the exact benchmark-wide share of 61.2%.

W2 and Q3 (ii): How reliably can C2 be distinguished from C7, and what do the disagreements look like?

Response: C2 and C7 are defined by different departures from the task (Table 2, L810–841). C7 keeps the original definitions and requirements and gives a verifiable attempt in which a key fact, computation, proof, or construction is wrong. C2 instead changes the meaning of a term, rule, or formal model, so that the task appears solvable under a different definition. In practice, agreement on this boundary is high. The two annotators agree on 34 of the 39 cases (87%) that both placed in C2 or C7, above their 78.4% agreement over all eight categories. On cases that both the judge and an annotator placed in C2 or C7, the judge agrees with annotator A on 30 of 31 and with annotator B on 29 of 30.

In the revision, we will add this study, the agreement statistics, and examples of disagreement to the appendix, and report the aggregate shares in the main text with the fine-grained categories as a descriptive breakdown.

Dear Reviewer kE1o,

We are grateful for the reviewer's detailed and constructive feedback. The suggested experiments on the wording of the completion question, the separation of the two pressure clauses, and the human annotation of failure categories directly strengthened the paper. We have run all three.

The revised paper reports the C1 and non-C1 contributions for all 19 configurations in Table 2. It adds the stricter completion question in Appendix E.4, the two single-clause pressure conditions in Appendix F.5, and the independent human annotation in Appendix C.2. We would appreciate hearing whether these results address the points raised in the review. We remain available for any further questions during the discussion.

Best regards,

Authors

Reviewer_xSHQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces NoGoBench, a benchmark of 1,061 scientifically impossible tasks and minimally edited feasible counterparts across six domains. Its invesitgation revolves around whether a model can detect something is impossible and execute it accordingly. They see a impossibility detection-execution gap, which indicates that models often executes or (claims to) perform a scientific task that it itself recognizes as impossible/infeasible. Authors also state that models detect impossibility with high Macro-F1 score (i.e. both feasible and infeasible tasks are detected correctly to a large extent). Lastly, they observe, increasing reasoning depth or post-training recipes to do not provide a significant solution to this problem.

Critical strengths
  1. I think the contributed NoGOBench of scientifically impossible tasks (and their possible counterparts) is interesting, alongwith the SCL library.
  2. Contrastive benchmark design is useful and the contrastive Macro-F1 seems a well motivated metric. So is cFCR.
  3. Preliminary analysis of the models based on the metrics and the goal of discovering the identification-execution gap is extensive. Authors have done some experiments on scaling reasoning and post-training as well. -- I appreciate the figures, tables and the effort that has gone behind it.
Critical weaknesses
  1. Some human validation has been carried out.

    i. But, it does not give me confidence about the entire benchmark.

    ii. There are concenrs about the SCL. For example, many theorems or many propositons are contigent upon having no proof "till date", which means later some of the propositions may be refuted. Its unclear how such cases are handled. Were they encountered at all? If it were, how did they do.

    iii. Human validation part has not been mentioned before the appendix. This seems to be a serious issue.

  2. Limited generalizability: Scientific impossibility may not adequately represent real-world agent constraints and value conflicts. The motivation and the dataset construction does not match (see questions)

  3. Simple experiments such as prompting to check impossibility first and then execute, should be eable to solve this problem. If this is easy to solve, then why should the community pay attention to the problem.

  4. Lastly, the authors do not clarify the list of contributions. The benchmark is an understandable contribution. But, is there a claim that the contrastive Macro-F1 and cFCR are novel? Or, there is no such claim. As a reviewer, it makes it very difficult to judge whether to weigh them as separate claims or not.

Questions for authors

Some targeted questions (line by line. Some are elaborations and have connections to the weaknesses)

  1. L060: Values are often abstract. Verifiability of such abstract rules might be subjective, even have socio-cultural connotations. Authors may re-think this connection to verifiable constraints, given this seems to be a motivation.
  2. L076: The contributions are not mentioned clearly. What are the claims? For example, NoGoBench is introduced. But, are you claiming the explicit feasibility judgment and direct execution metrics as claims as well? Its unclear.
  3. L181: SCL seems to be an interesting resource. However, many theorems or many propositons are contigent upon having no proof till now, which means later some of the propositions may be refuted. How do you handle such cases? Can you give clear examples?
  4. L181-L193: Do you mean if GPT-5.5 do not indicate, there are not human review at all?
  5. Can you show that simple baselines do not work here? If they work, what is the argument behind this problem's importance.
3 comments
AuthorsOpenReview ↗

Response to Reviewer xSHQ (1/2)

W4 and Q2: What are the paper's contributions? Beyond NoGoBench itself, do you claim the evaluation of explicit feasibility judgment and direct execution, Contrastive Macro-F1, or cFCR as novel contributions?

Response: We thank the reviewer for raising this.

We do not claim Contrastive Macro-F1 as a new metric. It is the standard Macro-F1 computed on minimally edited pairs, a contrastive design that follows Gardner et al., 2020 (L199–201). We claim the paired protocol with cFCR as one methodological contribution, not each condition or metric separately. cFCR is the rate of completion claims over impossible tasks that the same model correctly judged impossible in a separate conversation (Sec. 4.2–4.3), so it measures consistency with the model's own judgment rather than correctness.

Our contributions are as follows.

  • We introduce NoGoBench, 1061 evidence-grounded scientifically impossible tasks with minimally edited feasible counterparts across six domains, together with the SCL that records the assumptions, applicability conditions, and sources of each constraint.
  • We propose the paired protocol, with cFCR as its central measure, for measuring judgment–execution consistency.
  • We quantify this inconsistency across 19 configurations, with Contrastive Macro-F1 between 92.2% and 97.8% and cFCR between 8.3% and 76.6% (L088–091), showing that strong constraint recognition does not guarantee consistent execution. Reasoning, post-training, and execution pressure affect this gap differently (L099–105).
  • We find that the most common failure is specification evasion, where models redefine terms, weaken requirements, add assumptions, or substitute goals to claim completion, accounting for about 61% of the judge-classified inconsistent cases (L093–098).

We will add this list to the end of the Introduction.

W3 and Q5: Would a simple baseline, such as asking the model to check feasibility before executing, solve the problem? If it does, why should the community pay attention to this problem?

Response: We thank the reviewer for suggesting this baseline. Our primary goal is to define and measure the judgment–execution gap in a controlled setting, not to propose a mitigation. Scientific constraints have explicit applicability conditions and externally checkable evidence (L068–075), so the same model can be asked whether a task can be completed and its answer compared with its execution of the identical task.

We tested the baseline on the two models used in Sec. 6.3. In one conversation, the model first answers the feasibility question and then receives the unchanged execution prompt and completion question. On the tasks with valid runs under both procedures, cFCR changes as follows.

ModelcFCR (standard)cFCR (judge-then-execute)
DeepSeek-V4-Flash24.1%8.8%
Qwen3.5-35B-A3B34.7%0.3%

The baseline is effective and nearly eliminates cFCR for Qwen3.5-35B-A3B. This also supports our view that recognizing a constraint does not by itself ensure that execution follows it. Under the standard protocol, both models judge nearly all of these tasks impossible yet still report many of them as completed, and cFCR drops sharply once the judgment is given in the same conversation. However, in agentic tasks, it is hard to first judge whether a task is feasible and then execute it under the same environment and actions, because the environment state and the agent's actions unfold step by step during execution.

We will add this experiment to the appendix and refer to it in Sec. 4.

AuthorsOpenReview ↗

Response to Reviewer xSHQ (2/2)

W1 (ii) and Q3: How does the SCL handle propositions that are not yet proven and may later be refuted, were such cases encountered, and can you give examples?

Response: We thank the reviewer for these questions. Such propositions were encountered. Each SCL entry records the epistemic status of its constraint, shown as "Status" in Figure 2. Of the 2072 entries, 404 are recorded as conjectures, open problems, or unresolved questions, such as the Riemann hypothesis, Goldbach's conjecture, and the P versus NP problem. None of them is used to construct a task. All 1061 tasks rest on entries recorded as theorems, definitions, laws, established results, or no-go results.

Q4: Do entries that the LLM does not flag for review receive any human review?

Response: The reviewer's reading is correct. Entries that pass the screening receive no per-entry human review during construction. The screening is designed to be conservative. Each entry is standardized using only the retrieved evidence, and any entry that is flagged for review or not rated high confidence is excluded rather than reviewed and kept (L187–191).

W1 (i): Is the benchmark reliable beyond the 200 human-validated pairs?

Response: We place human review on the final task pairs because an SCL entry matters for the benchmark only through the tasks it supports. In Appendix F.4 (L1122–1150), a single annotator assessed 200 pairs sampled across disciplines and types of unattainability. The annotator judged 194 original tasks strictly impossible, five conditionally impossible, one open or insufficiently supported, and none feasible, and judged all 200 counterparts feasible. Recorded statuses therefore do not guarantee that every generated task states all necessary assumptions. Across the full benchmark, the 19 evaluated configurations judge on average 1.46% of impossible tasks feasible, which we take as a supplementary check rather than independent validation. Moreover, cFCR counts only tasks that the same model judged impossible (Eq. 1), so a completion claim on such a task is inconsistent with the model's own judgment whatever the true label.

In the revision, we will report the key results of the human validation at the end of Sec. 3 and state there that no task uses an entry marked as a conjecture, open problem, or unresolved question.

W2 and Q1: How does the value-based motivation in the Introduction connect to a benchmark built on verifiable scientific constraints, and do scientific impossibilities represent the constraints that real-world agents face?

Response: We thank the reviewer for this question. We agree that values are often abstract and that judging whether they are satisfied can be subjective and culturally dependent. This is why we do not build the benchmark on value constraints. When a constraint is contested, a departure during execution is hard to separate from disagreement about what the constraint requires. Scientific constraints have explicit applicability conditions and externally checkable evidence (L068–075), and models judge feasibility reliably on them. A completion claim on a task the model judged impossible can therefore be read as an inconsistency of the model rather than a disagreement about the constraint.

The two settings share the structure of the conflict rather than its content. In both, a constraint defines what counts as valid completion and conflicts with the task goal, and the question is whether a constraint the model has recognized remains binding during execution (L719–726). The failures we observe take this form, as models redefine terms, weaken requirements, or substitute goals, and some report success after explaining that the task is impossible (L093–098). Scientific constraints are also a favorable case, since their applicability conditions are explicit and their evidence can be checked externally. Even so, cFCR ranges from 8.3% to 76.6% across the 19 configurations (L023–025), and we expect consistency to be harder to maintain when constraints are vaguer and depend on context.

We do not claim that our quantitative results transfer directly to value constraints or to multi-step agent tasks, and value constraints remain future work (L485). In the revision, we will bring this rationale from Appendix A into the Introduction and state the scope explicitly.

Dear Reviewer xSHQ,

We appreciate the time the reviewer has devoted to our manuscript. The review raised questions about the reliability of the data, the link between value constraints and scientific constraints, whether a simple baseline would solve the problem, and what the paper contributes. Addressing them has made the motivation and the scope of the paper clearer.

In the revision, the Introduction now lists the contributions and states the scope explicitly. The end of Sec. 3 reports the key results of the human validation. Appendix F.6 adds the judge-then-execute baseline. We would like to know whether our responses have resolved the reviewer's concerns. If any part still needs further evidence or explanation, we would be happy to provide it before the discussion period ends.

Best regards,

Authors

Reviewer_p23GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NoGoBench explores the gap between "constraint identification" and "constraint compliance" in LLM: when a model correctly judges a task as scientifically impossible, can it still follow the constraint when performing the task? This benchmark covers six scientific domains, including 1,061 sets of "impossible tasks" and their corresponding "achievable tasks." The study tested the models' "judgment" and "execution" capabilities through independent dialogue sessions and conducted quantitative evaluations. In the multi-model configuration test, the models generally showed strong discrimination capabilities, but their performance in "incorrect completion" varied significantly.

Critical strengths

(1) It addresses an important issue: that "knowing the constraint" is not equivalent to "complying with the constraint," providing an effective analytical perspective.

(2) The construction and evaluation process is rigorous: by setting "achievable tasks" as a control, it distinguishes whether the model is accurately judging feasibility or simply adopting a "general rejection" strategy; the test covers multiple domains and model series.

Critical weaknesses

(1) Limited innovation. LM Agents May Fail to Act on Their Own Risk Knowledge (Tang et al., 2025), Why Agents Compromise Safety Under Pressure (Jiang & Tang, 2026), and ImpossibleBench (Zhong et al., 2025). Similar phenomena have also appeared in these studies; the core findings of this study (such as pressure-induced error completion and the discrepancy between judgment and execution) largely reiterate the phenomena found in these past papers, and it is not clear what new insights they bring.

(2) The “Conditional Error Completion Rate” (cFCR) quantifies different types of failure. For example, after the model correctly explains why the task is not feasible, it claims to have been successfully completed (i.e., C1, accounting for 24.5% of inconsistent cases), which is essentially “reporting inconsistency” rather than “violation of constraints” in the strict sense; although Section 6 and Appendix E discuss the sensitivity of this indicator, the study ultimately only summarizes these very different behaviors into a single concept of “gap”.

Questions for authors

(1) From a conceptual perspective, how does the "gap" explored in this study differ from previous studies on "inconsistency between knowledge and action" or "constraint compliance"?

(2) For C1-type cases (i.e., the model explains the task as infeasible but claims it has been successfully completed), should it be considered a "constraint violation" or a "reported error"?

(3) Given that different types of failures respond differently to "reasoning input" and "execution pressure," is "consistency between judgment and execution" a unified characteristic or the result of multiple different mechanisms working together? This is an important question.

5 comments
AuthorsOpenReview ↗

Response to Reviewer p23G (1/2)

W1 and Q1: How does the gap studied here differ conceptually from prior work on knowledge–action inconsistency and constraint compliance, in particular Tang et al. (2025), Jiang & Tang (2026), and ImpossibleBench?

Response: We thank the reviewer for pointing us to these three works, all of which we cite in Sec. 2. The gap we study is the inconsistency between a model's recognition of a constraint and its compliance with the same constraint. Measured this way, the most common form of inconsistency is specification evasion. Among the 5400 inconsistent cases classified by an external judge, 61.2% involve redefining terms, weakening requirements, adding assumptions, or substituting goals so that the impossible task appears completed (L093–098, L371–375). Below we explain how our setting differs from each of the three works.

Tang et al. are closest to our work. The two designs differ in three ways. First, the object of judgment differs. Tang et al. assess the risk of a given action from a pre-collected run, either as an abstract description or within its trajectory. We judge whether the task itself can be completed, before any action exists and without a candidate solution. Second, the information differs. Their identification test sees the full trajectory, including the risky action and the tool observations that follow, while their execution test sees only the part before that action. Our judgment and execution receive identical task text. Third, what must be recognized differs. Their judgment concerns risk in a given action. Ours requires recognizing the implicit scientific constraint that makes the task impossible, which the task text never states, and execution is then evaluated against that same constraint.

Jiang & Tang study how agents compromise explicit hard constraints under pressure. The constraints are stated to the agent, and recognition is assessed from whether its reasoning acknowledges them. In NoGoBench the scientific constraint is not stated, and recognition is a separate feasibility judgment on the same task text. Pressure changes the size of the gap but does not define it. The gap already appears under the plain instruction "Complete the following task". Our pressure results in Sec. 6.3 are consistent with their findings, and we do not claim this effect as new.

ImpossibleBench, like other work on constraint compliance, evaluates whether the output satisfies a constraint. It sets up conflicts between coding specifications and unit tests and counts any solution that passes the tests as cheating, but it does not condition the cheating rate on a separately elicited recognition judgment. In NoGoBench the constraint is not stated in the prompt, recognition is measured separately, and there is no test suite to game.

We will clarify these differences in Sec. 2 and state our contributions explicitly in the Introduction.

W2 and Q2: Should C1 cases, where the model explains why a task is infeasible but then reports completion, count as constraint violations or as reporting errors, and should cFCR combine them with other failures?

Response: We thank the reviewer for this question. We agree that C1 is a reporting error rather than a constraint violation in the strict sense. Its execution response substitutes an impossibility explanation or refusal for the requested deliverable, yet the model still reports completion. Appendix C already places C1 in its own group. The headline cFCR combines it with the other groups, and we agree that the paper should make this distinction clearer.

We keep C1 in cFCR because cFCR is defined on the model's own completion report rather than on verified success (L249–250, Eq. 1). Without C1, substantial false completion remains. Across the 19 default configurations, non-C1 cases contribute 21.0 of the 27.8 percentage points of pooled cFCR (L859–863). In the three models checked in Appendix E, external judges confirm a first-turn completion claim for 94.2% of non-C1 cases (L971, Table 5).

In the revision, we will use the three group names consistently, avoid describing C1 or every non-C1 case as a constraint violation, and report the C1 and non-C1 contributions alongside cFCR for all 19 default configurations.

AuthorsOpenReview ↗

Response to Reviewer p23G (2/2)

Q3: Given that different failures respond differently to reasoning and execution pressure, is judgment–execution consistency a unified property or the result of several mechanisms?

Response: We thank the reviewer for this important question. cFCR measures a defined user-facing event, a completion claim in an independent execution conversation on a task the model judged impossible (L245–252, Eq. 1). The question prompted us to connect Sec. 6.1 (reasoning) and Sec. 6.3 (pressure), which together suggest two sources of this event.

Every NY case ends in a completion claim, but the first departure can occur in different places (Appendix C, L792–807). In non-C1 cases, it occurs in the execution response. In C1 cases, the execution response explains why the task cannot be done or declines it, so the departure occurs only in the report. cFCR is the sum of the two contributions. For DeepSeek-V4-Flash, moving from no thinking to high reasoning lowers cFCR from 60.1% to 24.1%. The non-C1 contribution falls from 51.6% to about 9.5%, while the C1 contribution rises from 8.5% to about 14.7% (L408–415). From standard to forced execution pressure, cFCR rises from 24.1% to 50.8%, and about 92% of that increase comes from C1 (L465–475). Reasoning thus mainly lowers the non-C1 contribution, while pressure mainly raises the C1 contribution. This explains the contrasting changes the reviewer identifies.

The relative contributions of the two sources also differ across models. Across the 19 default configurations, the C1 and non-C1 contributions are only weakly correlated (Spearman rho = 0.18, Pearson r = 0.05). The C1 share of classified NY cases is 8.3% for GPT-5 (Figure 4(a)) but about 87% for GPT-5.6-Luna (L377–378). Reporting the two contributions separately therefore shows which source dominates for each model.

The two sources describe where the departure appears in the observed response, not distinct internal mechanisms. In the revision, we will add this interpretation to Sec. 4.3 to connect Sec. 6.1 and Sec. 6.3, and add the correlation analysis to Appendix E.

Dear Reviewer p23G,

We thank the reviewer for the careful reading of our paper and for pointing us to closely related work. The review asked how our setting differs from prior studies of knowledge–action inconsistency, whether C1 should be counted with other failures, and whether judgment–execution consistency is a single property. These questions led us to state the scope of our contribution more precisely and to separate the two sources of false completion.

The revised paper now states the differences from Tang et al., Jiang & Tang, and ImpossibleBench in Sec. 2. It reports the C1 and non-C1 contributions for all 19 configurations in Table 2, and explains the two sources in Sec. 4.3. We would be grateful to know whether these changes and our responses address the reviewer's concerns. We would be glad to clarify any remaining point during the discussion.

Best regards,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Manuscript Uploaded

Dear Reviewers and Area Chair,

We thank all reviewers for their careful reading and constructive comments. We have uploaded a revised version of the paper that includes the changes promised in our responses. All revised text, including new tables and captions, is shown in blue, and the main text remains within the 10-page limit. Line numbers cited in our responses refer to the original submission.

The main changes are listed below.

  • Introduction. An explicit list of contributions, and the rationale for using scientific constraints together with a statement of scope.
  • Sec. 2. The differences from Tang et al. (2025), Jiang & Tang (2026), and ImpossibleBench.
  • End of Sec. 3. The key results of the human validation, and the statement that no task uses an entry recorded as a conjecture, open problem, or unresolved question.
  • Sec. 4.3 and Table 2. The two sources of false completion, with the C1 and non-C1 contributions to cFCR for all 19 default configurations.
  • Sec. 5.3, 6.1 and 6.3. Consistent names for the three failure groups, aggregate shares first, and the link between the reasoning and pressure results.
  • Appendix C.2. Independent human annotation of the failure categories.
  • Appendix E.3 and E.4. The correlation between the C1 and non-C1 contributions, and the stricter completion question.
  • Appendix F.5 and F.6. The two single-clause pressure conditions, and the judge-then-execute baseline.

Best regards,

Authors

AuthorsOpenReview ↗

Summary of the Rebuttal

Dear Area Chair,

As some reviewers may not yet have had the chance to respond to our rebuttal, we provide a brief summary of how it addresses the main concerns, in case it is useful to the Area Chair. Reviewer kE1o found that the new results strengthen the paper and kept the positive rating. All changes are included in the revised paper.

  • Relation to prior work (Reviewer p23G). The gap we study is the inconsistency between a model's recognition of a constraint and its compliance with the same constraint. Tang et al., Jiang & Tang, and ImpossibleBench were already cited in Sec. 2 of the original submission, and the revised Sec. 2 states more explicitly how our setting differs from each.
  • Interpretation of cFCR (Reviewers p23G and kE1o). C1 is now reported separately. Non-C1 cases account for 21.0 of the 27.8 percentage points of pooled cFCR, and the gap is reduced but persists under the stricter completion question suggested by Reviewer kE1o.
  • Pressure confound (Reviewer kE1o). Separating the two clauses of the forced prompt shows that the demand for a solution raises cFCR more than the instruction to confirm completion.
  • Failure taxonomy (Reviewer kE1o). On all 97 NY cases of Claude Opus 5, two independent annotators agree on C1 versus non-C1 for 95.9% of cases, and their shares of specification evasion are close to the judge's.
  • Data validity (Reviewer xSHQ). No task uses an entry recorded as a conjecture, open problem, or unresolved question, and in the human review, 194 of 200 original tasks were judged strictly impossible and none feasible.
  • Simple baseline and contributions (Reviewer xSHQ). Judging feasibility before executing in the same conversation sharply lowers cFCR, which supports our finding that recognizing a constraint does not by itself ensure compliance. Extending this baseline to agentic tasks is harder as states and actions unfold during execution, while our controlled design enables direct comparison of judgment and execution on the same task text. The Introduction now lists the contributions and explains why we use scientific rather than value constraints.

We respectfully ask the Area Chair to consider our responses and the revised paper when making the decision.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.