- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Tasks given to an AI agent can be complex and varied. It is not unusual that a task specification in ambiguous, misses key details and requirements, or is otherwise infeasible. The authors argue that our understanding of AI behaviour on those instances is sub-optimal and propose a benchmark to measure performance on selected infeasible tasks. To correct sub-optimal behaviour the authors then propose to intervene on their harness (defined as prompts, processors, control flow, and stopping behavior).
Critical strengths
Acting under uncertainty is a difficult and important problem. I commend the authors for tackling and analyzing it. Unfortunately, clear strengths were difficult for me to surface from the main body of the paper, in my view (as a non expert in this particular sub-field of AI agents and benchmarking) because the contribution wasn't sufficiently contextualized in the context of related work and current issues with AI agents.
Critical weaknesses
Although the authors consider a problem that is reasonable (what should AI agents do given infeasible tasks), it is not clear to me that current AI agents underperform or fail in interesting ways on these kinds of problems. In my view, the paper is missing some evidence for the problem they seek to correct, such as a clear example where the failure is apparent.
Further, for someone not familiar with Terminal-Bench, it is difficult to fully grasp the kind of examples the authors propose in their benchmark without examples. Without these elements it is difficult for a reader to appreciate what problem is being solved and what correct behaviour should look like.
More justification for their harness choices would be useful to better contextualize the contribution. Simple baselines, like prompt variations, would be useful.
Questions for authors
- The authors suggest that AI agents should abandon infeasible tasks. However, useful output can often be provided to the user, e.g., solving a closely related task, explaining why the task is infeasible and what feasible variations exist, etc. A discussion of practical AI use in these scenarios would be appropriate.
- Are there simple baselines that could be instantiated for this problem, e.g., appending to the prompt "Analyse the feasibility of the task and report missing requirements where appropriate"? In their related work, the authors list several related benchmarks that involve uncertainty in decision-making, could their methods be used on these benchmarks? What about the proposed benchmark is missing in related work?
- Minor: you define an agent as a tuple (M, H). What are M and H? (This is made apparent later in the paper, but should be described when first defined)
- Could the authors discuss the question of acquiring missing information in an on-going task? For example, if there is a missing requirement, an agent might search the user's files or codebase to make an informed guess and nevertheless complete the task the user most likely wanted solved.
3 comments
Response to Reviewer Czag (2/3)
Response to Reviewer Czag (2/3)
Q2. What the blocked tasks look like and what correct behavior is. Terminal-Bench tasks ask agents to complete work using files and tools. The three blocked families differ in why completion is unwarranted and in when the decisive evidence appears (Table 1). Simplified illustrations of each family:
- Infeasible. A prerequisite is absent or the requested operation is unsupported. Example: recover eleven records when the log holding six of them is missing (Q1). The agent should identify the missing input and defer rather than invent the data.
- Ambiguous. Requirements conflict or do not specify a warranted completion. Example: two mandatory instructions require the same section to be both deleted and preserved unchanged. The agent should surface the conflict and ask for revised requirements rather than silently choose one and claim completion.
- Trap. An input looks usable until execution reveals a latent blocker. Example: an archive supplied for extraction turns out to be irrecoverably corrupted, with no replacement copy. Initial inspection is appropriate; once the blocker is found, the agent should identify it and abstain.
Warranted deferral requires an agent-authored certified abandon that names the correct blocker. The original feasible source tasks remain should-act controls: with usable inputs and compatible requirements, the agent should execute the task and satisfy the native verifier (Secs. 2.1–2.2).
Q3. What this adds to related benchmarks, and whether the methods transfer. We agree that concurrent work overlaps with ours in different ways: AgentAbstain evaluates paired act/abstain correctness, Agentic Abstention emphasizes timely stopping, and HiL-Bench evaluates selective help-seeking. Our benchmark adds an explicit diagnostic of whether task-grounded feasibility judgments govern behavior. It separately measures recognition, non-fabrication, agent-authored certified abandon, and correct justification, alongside belief and reconsideration probes. Native-verifier evaluation of the feasible counterparts checks that improved deferral preserves useful execution. HiL-Bench obtains missing information through targeted questions so that execution can continue; our blocked variants test warranted deferral when the original request remains unsupported under the available conditions.
Our evaluation and harness-search methods can be adapted to these benchmarks wherever blocker metadata, authenticated trajectories, and suitable completion checks are available. The warranted action and certification rules must match each setting; in HiL-Bench, for example, the warranted action is targeted help-seeking followed by execution. A two-sided objective could then reward both appropriate responses to blockers and successful feasible-task execution. We instantiate the approach in Terminal-Bench and retail τ²-bench; applying it to the other benchmarks is a natural extension.
Response to Reviewer Czag (3/3)
Q4. Why harness interventions, beyond simply asking the agent to consider feasibility? Feasibility-aware prompting is a reasonable starting point, and our formulation does not require harness search. We use the harness as a broader search space for how the agent checks and constrains its own behavior: besides prompts, it includes processors, control flow, and stopping behavior (Sec. 2.4). These components determine when feasibility is assessed, how new evidence triggers reassessment, and how the judgment governs execution or certified abandon, while the model stays fixed.
A small matched retail experiment illustrates the difference (Table 21). On the same 32 held-out pairs, strict SA is 9/32 for the initial harness, 7/32 when we add a feasibility instruction to it, and 11/32 for the joint-selected harness. Feasible solve is 14/32, 16/32, and 24/32, and false abstention is 0/32 for every arm. The processor architecture and assessment protocol are held fixed, so this compares guidance conditions only and does not isolate the benefit of non-prompt components. With one rollout per task, the results are descriptive.
Q5. Can agents give useful output instead of simply abandoning? We agree that deferral should come with useful assistance, and our formulation does not ask agents to abandon silently. Deferral means declining to claim completion of the specified request once the evidence shows that it cannot be completed. The protocol asks the agent to give an evidence-based rationale for its feasibility judgment. That explanation helps the user understand the blocker and decide whether to supply more information, revise a requirement, or pursue a feasible alternative.
Our metrics check whether the agent identifies the correct blocker and states the correct reason (M0 and M3; Sec. 2.3 and App. A.2). They do not measure how useful the rationale is. Evaluating that usefulness, for example with finer-grained or judge-based scoring, is a natural next step. Sec. 2.2 now describes deferral as the honest status of the original request, which can be followed by an explanation, labeled partial results, or feasible alternatives.
Q6. Should agents acquire missing information from the user's files or codebase? Yes, agents may gather information from the environment. Sec. 2.1 defines t* as the first point at which decisive feasibility evidence becomes available, possibly after tool calls; Table 8, for example, includes an ownership conflict revealed by read-only order and profile checks. If searching authorized files or a codebase resolves a missing requirement, the task is feasible under our definition, and proceeding is correct. Our blocked variants are validated so that the missing piece cannot be recovered from the environment, and retail blockers cannot be resolved by asking the simulated user (App. A.1.3).
Q7. Defining (M, H). Thank you for pointing this out. M is the fixed language model and H its execution harness (prompts, processors, control flow, and stopping behavior); together they form the agent. Both were introduced only in the Figure 1 caption and Sec. 2.4, and they are now defined where the tuple first appears (Sec. 2.1).
We hope the added context makes the contribution clearer and would be glad to answer further questions before the discussion closes.
Response to Reviewer Czag (1/3)
Thank you for recognizing the importance of acting under uncertainty. Changes in the revised PDF are in blue.
Context: the problem we address. Agent benchmarks such as Terminal-Bench and SWE-bench reward task completion, so an agent that fabricates a result on an impossible task can look successful, while abstention-only evaluations invite refusing feasible work. Recent benchmarks (AgentAbstain, Agentic Abstention, HiL-Bench) study when agents should stop or ask. We test whether recognizing a blocker actually governs behavior, and we separate the agent's behavior from credit produced by the evaluation setup. Three findings follow:
The introduction and related work now state this positioning.
Q1. A clear example of an agent failing on a blocked task. The following comes from a GPT-5.5 Terminal-Bench run in the bare-agent condition A0 (Table 11).
Those checks validate the records the agent generated; they do not show that the missing data were recovered. The agent observes the missing prerequisite and still produces an unsupported completion. This A0 example is separate from the 64-task protocol-baseline diagnostic in Sec. 3.2.