acceptodds
Under review as a conference paper at ICLR 2027

HandsOffBench: Can Agents Autonomously Complete Everyday Projects from Start to Finish?

Abstract

Large language models are increasingly deployed as general-purpose assistants to fulfill everyday user requests across applications and services. As their performance on individual tasks improves, a central question is whether agents can reliably complete entire everyday projects without further human intervention. We introduce \ours, a benchmark comprising 32 everyday projects in a dynamic simulated world featuring 70 web applications, 195 stateful services, and code execution environments. Collected from individuals with diverse backgrounds, these projects are adapted into executable tasks with carefully designed dependencies grounded in their practical requirements. Later stages build on earlier decisions, computed values, and generated artifacts, forming extended workflows with end-to-end runs spanning thousands of steps. We combine state checkers with rubric-based verifiers to assess subtask outcomes from the final environment state and generated artifacts, using an average of 105 scoring criteria per project. Shared criteria support comparisons of the same subtasks in isolated and end-to-end execution. We evaluate frontier models with a minimal scaffold and with widely used agent harnesses. The strongest evaluated model completes only 20% of projects with the minimal scaffold and 30% with harness support. On the subset evaluated in both settings, agents perform substantially worse on the same subtasks within complete projects than in isolation. These findings highlight the gap between subtask competence and reliable end-to-end completion of everyday projects, positioning \ours as a testbed for advancing model capabilities and agent harness design.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yVMCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HandsOffBench, a benchmark for testing whether LLM agents can finish everyday projects on their own, without a human stepping in. Each project has a user request, a starting environment, and specific evaluation criteria, and is broken into dependent subtasks that form a DAG. Tasks are running in a simulated world with locally hosted apps and stateful services. Some also include dynamic events that change information mid-execution. Evaluation looks at the final environment state and generated artifacts, using state checkers plus an LLM verifier. The experiments compare several model families under a minimal scaffold, and also test Codex and Claude Code as harnesses. The main finding is that no agent fully completes any project, and the best setup only reaches around 50% fine-grained score.

Critical strengths
  1. The benchmark introduces dynamic events into long-horizon projects. These events change information or supersede existing records during execution, requiring agents to reassess earlier decisions and update dependent results. This adds a realistic dimension that static task suites often lack.

  2. Adequate Experiments.

Critical weaknesses
  1. The paper analyzes long-horizon behavior in Section 4.3, but this analysis leaves an important ambiguity unresolved. It is unclear whether the observed difficulty stems from long-range coupling between subtasks, or simply from having more subtasks and a longer task chain, which could lead to compounding errors over extended execution. The isolated-versus-end-to-end comparison shows that the same subtask scores higher in isolation, but this comparison changes several factors at once: prerequisite establishment, context length, dependency tracking, and error accumulation. As a result, it does not isolate the effect of long-range coupling itself. To clarify this, I would suggest an ablation that selects a subset of projects and starts execution from gold-standard prerequisite states, while keeping the full project structure, subtask set, and execution budget. If agents still struggle to complete the project under these conditions, the result would provide more direct evidence that the difficulty is due to long-range coupling rather than task length or subtask count alone. If performance improves substantially, then the current claim about long-horizon difficulty would need to be reconsidered.
Questions for authors
  1. Could you add an ablation that separates long-range coupling from task length? For example, running a subset of projects from gold-standard prerequisite states while keeping the same subtasks, project structure, and budget would help clarify whether the difficulty comes from coupling itself or simply from longer task chains.

  2. Could you provide more detail on how the projects were constructed? Specifically, how were subtask dependencies designed, how were prerequisite states defined, how were dynamic events inserted, and how were the rubrics sets built?

1 comment

Dear Reviewer, thank you again for your valuable feedback. With the discussion period closing in less than one day, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. We are happy to discuss any further questions. Thanks for your time and consideration.

Reviewer_F5tvOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents HandsOffBench, a benchmark comprising 32 everyday projects in a dynamic simulated world featuring 70 web applications, 30 stateful services, and code execution environments; collecting from individuals with diverse backgrounds, these projects require coordinating multiple subtasks across applications and services. Their experiments show that frontier models with harnesses can achieve only around 50% score; highlight the gap between subtask competence and reliable end-to-end completion of everyday projects. But, overall, in the current stage, the paper looks very under-developed and rushed; and crucial details on motivation, experiments, evaluation, error analysis are missing.

Strength:

  1. Introduced HandsOffBench, a benchmark comprising 32 everyday projects across diverse domains and a total of 388 subtasks; with experiments on different frontier models and scaffolds.
Critical strengths

Introduced HandsOffBench, a benchmark comprising 32 everyday projects across diverse domains and a total of 388 subtasks; with experiments on different frontier models and scaffolds.

Critical weaknesses

Weakness:

  1. The problem statement and the research question is not properly established. In line 44, it says the paper asks if the models and agents can sustain competence throughout an entire project without further human intervention. But, it is not clear what type of things/tasks/interventions the authors are considering “human intervention”?
  2. Related works are not adequately explored. Some very common works like ClawBench, REALM-Bench, and some more recent works are not explored and compared properly. Also, it is not clear how this benchmark is different from existing ones;
  3. There is no single example provided in the main paper, no excerpt, nothing - to check, evaluate and see if the tasks are developed properly or get an idea of it.
  4. Implementation details are incomplete. For open models, do you use open-weights or use API calls? Also, no details are given on the machine. Also, it is also not clear if the tasks require additional software or tools.
  5. Throughout the paper, there are a lot of subjective terms and sentences used, without any proper evidence. Such as, line 319-321. Such qualitative remarks should be made with respective figures/data showing such errors.
  6. The paper mentions HandsOff scaffold, but no proper details are given on how it is developed, what things are available and more. Existing literature already established that harness/scaffold design can make or break agentic evaluation setups.
  7. This paper had a chance to do extensive analysis on error patterns and show where, how and why - models are failing and then suggest a roadmap to improve on this area; but currently, it lacks the most important part the paper should have. The paper does report some numbers, but no table or figure or supporting information is provided to go deep and check the claims/any pattern.
Questions for authors

Question: Following up on the weaknesses, my questions are:

  1. Regarding W1, can you please clarify human intervention and expected behavior properly, maybe with an example; and establish the motivation and the problem?
  2. Regarding W2, can please explore more recent works, providing comparison data with existing works more explicitly.
  3. Can you please add a clear example from the benchmark in the main paper? Also, provide more information on appendix or supplementary data. Please atleast provide some qualitative error/execution analysis, showing an error made by an agent, and how if there were a human intervention, it could have avoided that; and how it would have affected the overall completion scores?
  4. Can you please provide all the implementation details as necessary?
  5. Please share what % of projects the agents managed to complete properly? Is it zero?
  6. Can you please provide full details on the HandsOff scaffold?
  7. Can you please do an extensive qualitative and quantitative error analysis and provide insights for future development based on these insights?
Reviewer_diY8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces HandsOffBench, a benchmark of 32 everyday projects (388 interdependent subtasks) in a simulated world with 70 web applications and 30 stateful services, some with dynamic events. Outcomes are scored against about 195 fine-grained criteria per project. The best configuration scores about 50% with no project fully completed, which the authors attribute mainly to long-horizon execution.

Critical strengths

A timely question: end-to-end project completion rather than isolated operations, complementing existing SWE, web, and OS benchmarks. Substantial engineering effort, with fine-grained criteria that enable partial credit and failure diagnosis. The paired isolated versus end-to-end comparison is a sensible way to probe horizon effects.

Critical weaknesses

Evaluation validity is unverified. There is no human baseline, solvability check, or human agreement study for the Claude-Opus-5 verifier, a concern since Claude-Opus-5.5 ranks first. All results come from single runs without confidence intervals, despite many differences under two points. The dynamic events, a key selling point, are never analyzed.

Some inconsistencies in the paper, could be bad proof-reading/or unclear text. #1 Table 2 reports tokens in millions, while the text says thousands. Section 3.2 lists 194 services versus 30 elsewhere. Section 4.1 mentions GLM, but no GLM results are reported; the abstract claims the best model "completes only 46.23% of projects," yet Section 4.2 states that no project is fully completed; the figure is a criteria-satisfaction rate

Questions for authors

Evaluation reliability is unverified, and several reported statistics are inconsistent. Clarify discrepancies in understanding and/or fix text.

1 comment

Dear Reviewer, thank you again for your valuable feedback. With the discussion period closing in less than one day, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. We are happy to discuss any further questions. Thanks for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.