- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HandsOffBench, a benchmark for testing whether LLM agents can finish everyday projects on their own, without a human stepping in. Each project has a user request, a starting environment, and specific evaluation criteria, and is broken into dependent subtasks that form a DAG. Tasks are running in a simulated world with locally hosted apps and stateful services. Some also include dynamic events that change information mid-execution. Evaluation looks at the final environment state and generated artifacts, using state checkers plus an LLM verifier. The experiments compare several model families under a minimal scaffold, and also test Codex and Claude Code as harnesses. The main finding is that no agent fully completes any project, and the best setup only reaches around 50% fine-grained score.
Critical strengths
-
The benchmark introduces dynamic events into long-horizon projects. These events change information or supersede existing records during execution, requiring agents to reassess earlier decisions and update dependent results. This adds a realistic dimension that static task suites often lack.
-
Adequate Experiments.
Critical weaknesses
- The paper analyzes long-horizon behavior in Section 4.3, but this analysis leaves an important ambiguity unresolved. It is unclear whether the observed difficulty stems from long-range coupling between subtasks, or simply from having more subtasks and a longer task chain, which could lead to compounding errors over extended execution. The isolated-versus-end-to-end comparison shows that the same subtask scores higher in isolation, but this comparison changes several factors at once: prerequisite establishment, context length, dependency tracking, and error accumulation. As a result, it does not isolate the effect of long-range coupling itself. To clarify this, I would suggest an ablation that selects a subset of projects and starts execution from gold-standard prerequisite states, while keeping the full project structure, subtask set, and execution budget. If agents still struggle to complete the project under these conditions, the result would provide more direct evidence that the difficulty is due to long-range coupling rather than task length or subtask count alone. If performance improves substantially, then the current claim about long-horizon difficulty would need to be reconsidered.
Questions for authors
-
Could you add an ablation that separates long-range coupling from task length? For example, running a subset of projects from gold-standard prerequisite states while keeping the same subtasks, project structure, and budget would help clarify whether the difficulty comes from coupling itself or simply from longer task chains.
-
Could you provide more detail on how the projects were constructed? Specifically, how were subtask dependencies designed, how were prerequisite states defined, how were dynamic events inserted, and how were the rubrics sets built?
Dear Reviewer, thank you again for your valuable feedback. With the discussion period closing in less than one day, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. We are happy to discuss any further questions. Thanks for your time and consideration.