acceptodds
Under review as a conference paper at ICLR 2027

AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

Abstract

Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents’ responses to these information gaps remain poorly characterized across models and execution harnesses. We present AM-Bench, a benchmark built from 48 tasks adapted from Terminal-Bench 2.0 and Harness-Bench. Each task includes a fully specified reference and six variants spanning ambiguous or missing goals, context, and constraints. We evaluate every pairing of five models and four harnesses over 20,160 runs. Our results reveal distinct execution deficits: goal gaps hinder completion, context gaps increase token use, and constraint gaps weaken compliance without necessarily preventing completion. Missingness lowers success more than ambiguity in all three domains. These effects also reshape system comparisons: advantages under complete information need not persist, and rankings depend on model × harness pairings and the evaluation criterion—completion, success, or repeat consistency. Estimates based on observed clarification requests and matched fully specified runs further suggest that clarification could recover unfinished work while leaving most constraint violations unresolved. Together, these findings motivate joint model–harness evaluation across information conditions and targeted clarification of unresolved requirements before dependent actions.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Task Underspecification, Agent Benchmarking, Model–Harness Interaction, Clarification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_izP8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper introduces a new benchmark that adapts two existing benchmarks (Terminal-Bench and Harness-Bench) into goal, context, and constraint (permitted actions) variants to gauge how different information gaps affect model-harness pairings. The empirical results motivate model-harness evaluation across different conditions of task underspecification.

Critical strengths
  • The empirical results demonstrate that a model’s performance can degrade depending on the harness it’s paired with.
  • The benchmark isolates completion (ability to finish a task) from compliance (ability to follow rules) and demonstrates that when constraints are missing, agents frequently finish the task but violate rules in the process.
Critical weaknesses
  • When an agent asks a clarifying question during an underspecified task, the benchmark does not actually provide an answer, but the run’s outcome is instead replaced with the agent’s average success rate on the task's fully specified version. This approach does not measure the agent’s actual continuation of the task and likely creates an overly optimistic estimate of clarification.
  • The should report confidence intervals, especially when making comparative claims based on small pp differences (e.g., Codex has a 1.5 pp Success lead over OpenClaw, in Table 21) given the high variance inherent in binary success metrics evaluated over only three runs per configuration.
Questions for authors
  • The authors should explain more in the main text how AM-Bench differs from UnderSpecBench and LHAW. What is “benign task underspecification” and how does it differ from UnderSpecBench?
  • The underspecified condition definitions (Table 1) would be easier to interpret if a running example (e.g., from one of the benchmark tasks) was provided.
  • The Task conditions (GA, GM, etc.) should be defined in the main text with examples, like in Appendix B.2. Otherwise, it is difficult to interpret Table 2 and Figures 3 and 4.
Reviewer_9GwqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AM-Bench tests how agents behave when goals, context, or constraints are unclear or missing, across different models and agent frameworks. They adapt 48 tasks from Terminal-Bench 2.0 and Harness-Bench, using an LLM and author review to create six underspecified versions of each. They evaluate five models across four frameworks, measuring completion, compliance, success, cost, and clarification requests. Their key finding is that systems performing best with clear instructions may not remain best when information is missing. However, all versions are graded against the fully specified requirements, so failures can reflect unknowable requirements rather than agent mistakes. Clarification questions receive no answers; their potential benefits are estimated from fully specified runs.

Critical strengths

The direction is timely

Critical weaknesses

The key methodological limitation of the work is that - the authors although generate perturbed versions of existing terminal bench / harnessbench tasks (by making them underspecified) - they keep the evals same.

After making the tasks underspecified or perturbing it such that multiple correct solutions exist, makes the original eval unapplicable. And that severely harms the reliability of the results proposed. I request the authors to clarify this in detail. Given that the dataset only has 48 tasks, if possible, i request them to directly share them in their rebuttal.

Questions for authors

see weakness

Reviewer_53ngOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Overall, this paper addresses a very relevant problem (joint model–harness evaluation). The benchmark’s most useful idea is to compare the same intended task under different information conditions. AM-Bench applies this design to 48 tasks across five models and four harnesses, examining how ambiguous or missing goals, context, and constraints affect completion, compliance, and clarification behavior. The paper reports that goal gaps reduce completion, context gaps increase token use, and constraint gaps substantially reduce compliance while preserving completion.

The study provides useful evidence that performance under fully specified tasks does not necessarily carry over to underspecified settings. My main concern is how we interpret these outcome gaps: failure against the intended reference does not automatically imply an unreasonable decision given the information available to the agent. Some construction choices also complicate attribution of the observed effects to underspecification alone.

Critical strengths
  • Same-task comparisons across information conditions: Holding the intended task fixed provides a useful reference for measuring outcome gaps. This is more informative than comparing unrelated fully specified and underspecified tasks, where task difficulty could explain the differences.

  • Joint model–harness evaluation: The complete cross shows that conclusions depend on the particular pairing. This strengthens the benchmark’s practical value and cautions against drawing broad conclusions from a model evaluated through only one harness.

  • Separate completion and compliance measures: The distinction captures an important aspect of agent behavior: producing the requested result does not establish that the agent respected the intended boundaries. The outcome decompositions and repeated-run analyses help explain these differences.

  • Useful combination of existing evaluation directions: The contribution brings together controlled information conditions, model–harness comparisons, and clarification analysis. I see value in this combined design, even though its individual components have close precedents.

Critical weaknesses
  • The interpretation of outcome gaps could be clearer. Comparing the same intended task across information conditions is a useful design. However, lower reference-task completion does not necessarily indicate an unreasonable decision given the agent’s observations. In the file-summary example, an alternative summary can reasonably satisfy the ambiguous request, and asking for the objective is appropriate when the goal is missing.
The paper acknowledges this distinction, but I would like it carried more consistently into the interpretation of system rankings and failure rates. Clearly separating reference-task attainment from decision appropriateness given the available information would help readers understand what the benchmark establishes. An additional appropriateness assessment on a representative subset would strengthen this interpretation.

  • The constraint results need more precise attribution. In the scheduling example, Constraint–Missing removes the prohibition, changes tool descriptions, and encourages the agent to use potentially useful readiness or validation material. The resulting contrast therefore combines restriction omission with changed action incentives and reduced visibility into sensitive access.
This is a meaningful stress test, but it limits how directly the observed compliance drop—including the 0% Success result—can be attributed to missing constraints alone. I encourage the authors to describe this combined intervention more explicitly in the main text and qualify the corresponding conclusions. A small controlled ablation separating restriction presence from encouragement, while holding other tool information fixed, would provide stronger evidence for the proposed explanation. This is the concern most likely to affect the substantive interpretation of the results.

  • The clarification contribution would benefit from a sharper statement of scope. The observed request rates provide useful evidence about when agents ask, particularly the contrast between missing goals and missing constraints. The Full-reference estimates also offer a common basis for examining potential recovery opportunities.
However, these estimates do not establish whether the actual questions are sufficient, timely, or successfully used after an answer. Although the assumptions are documented, this distinction should remain explicit wherever the paper discusses clarification gains. A modest validation using answered continuations would strengthen this part substantially. Without that validation, the contribution remains useful as an analysis of request behavior and Full-reference recovery opportunities.

  • The contribution relative to prior work could be stated more directly. The paper already acknowledges relevant work on controlled underspecification, model–harness evaluation, boundary violations, and interactive clarification. Its distinctive contribution is the combination of these perspectives within grouped information conditions and a complete model–harness cross.
I would encourage the authors to foreground this contribution and explain which findings specifically depend on the combined design. This would make the novelty easier to assess and give readers a clearer account of what AM-Bench adds to existing benchmarks.

Questions for authors
  • Can you separately evaluate reference-task attainment and decision appropriateness given the agent-visible information? How would this affect comparisons involving reasonable alternative interpretations and appropriate clarification?

  • Can you isolate the effects of removing constraints, changing tool descriptions, and encouraging access to readiness or validation material? Does the Constraint–Missing result persist without the added encouragement?

  • For hidden goals and restrictions, how do you establish whether the intended behavior is inferable from the remaining information? Would careful human annotators agree on the appropriate action?

  • Which findings specifically require the complete model–harness cross, and what additional insight does the Full-reference analysis provide alongside existing interactive clarification benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.